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PREFACE 


This  document  reports  the  work  performed  by  the  Institute  for  Defense  Analyses 
for  the  Office  of  the  Under  Secretary  of  Defense,  Acquisition,  Technology,  and  Logistics 
(Joint  Advanced  Concepts)  in  partial  fulfillment  of  the  task  entitled  “Improving  DoD 
Acquisition.” 

The  authors  wish  to  thank  Dr.  John  Hanley,  Mr.  Michael  Fitzsimmons,  and  the 
reviewers,  Dr.  David  Graham  and  Dr.  Stuart  Starr,  for  valuable  comments  and 
suggestions. 
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SUMMARY 


The  Concept  Decision  (CD)  process  was  launched  by  the  2006  Quadrennial 

Defense  Review  (QDR)  of  the  Department  of  Defense  (DoD),  which  states  that 

Assessments  of  potential  solutions  should  be  informed  by  the  acquisition 
community’s  judgment  of  technological  feasibility  and  cost-per- increment  of 
capability  improvement,  and  by  the  resource  community’s  assessment  of 
affordability.  These  inputs  will  be  provided  early  in  the  decision-making  process, 
before  significant  resources  are  committed. 

Under  the  direction  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics;  the  Director,  Program  Analysis  and  Evaluation;  and  the  Vice  Chainnan  of  the 
Joint  Chiefs  of  Staff  (the  Tri-Chairs),  the  CD  process  is  designed  to  better  harmonize  the 
Department’s  three  major  decision-support  processes — requirements  generation, 
acquisition  management,  and  resource  allocation.  The  benefits  are  expected  to  include: 

•  Greater  future  stability  of  acquisition  programs  that  have  been  subjected  to  the 
early  CD  process 

•  Faster  development  and  fielding  of  new  military  systems 

•  More  explicit  consideration  of  potential  non-materiel  solutions  to  important 
capability  gaps,  and 

•  The  analytic  identification  of  programs  for  potential  divestment  to  serve  as 
sources  of  funding  for  added  capabilities  from  within  the  same  general 
mission  area  or  portfolio. 

The  purpose  of  the  Concept  Decision  process  is  to  make  an  early  decision  on  an 
affordable  capability  investment  strategy  responsive  to  future  joint  warfighter  needs.  The 
measure  of  success  is  the  creation  of  sufficient  information,  in  a  portfolio  based  context, 
to  make  such  investment  decisions  so  as  to  yield  predictable  performance.  Four  pilot 
programs  were  nominated  to  develop  an  experiential  base  from  which  implementation 
guidance  can  be  constructed.  To  date  the  CD  process  has  focused  primarily  on  investment 
in  equipment  -  in  part  because  the  initiative  has  been  led  by  the  office  of  the  Under 
Secretary  of  Defense  (Acquisition,  Technology  and  Logistics),  who  has  been  delegated 
decision  authority  on  such  matters.  However,  the  basic  portfolio  analysis  and 
management  concept  includes  all  elements  of  the  defense  program  that  contribute  to 
military  capabilities,  principally  equipment  and  force  structure.  The  extent  to  which  the 
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CD  process  will  actually  be  used  for  making  decisions  on  future  force  structure  has  not 
been  resolved. 

The  primary  purpose  of  this  paper  is  to  develop  an  approach  to  portfolio  analysis 
in  the  CD  process  and  to  guide,  in  a  general  way,  future  practitioners  in  undertaking  CD 
Evaluations  of  Alternatives  (EoAs).  The  basic  structure  of  such  analyses  is  to  identify 
zero-net-cost  changes  to  the  appropriate  portfolio  that  would  result  in  improved 
capabilities  to  meet  specified  challenges.  The  approach  to  portfolio  analysis  presented  in 
this  paper  is  as  applicable  to  the  tasks  of  the  Institutional  Reform  and  Governance 
(IR&G)  Capabilities  Portfolio  Managers  (CPMs)  (another  and  more  far-reaching 
initiative  of  the  2006  QDR,  described  later)  as  it  is  to  the  tasks  of  the  CD  EoA  Study 
Directors,  which  is  the  main  topic  of  this  paper. 

This  paper  explores  a  number  of  important  considerations  in  conducting  Concept 
Decision  Evaluation  of  Alternatives,  particularly  with  regard  to  the  key  process  of 
defining  and  analyzing  the  capability  portfolio  relevant  to  the  mission  area.  The  raison 
d’etre  of  a  Concept  Decision  review  is  to  address  a  critical  capability  gap  that  has 
emerged  through  other  processes  and  to  do  so  early  enough  so  that  the  resulting  decision 
can  be  integrated  into  the  Department’s  long  range  budget  plans  without  undue 
disruption.  The  normal  expectation  is  that  currently  unplanned  resources  will  be  required 
to  provide  capabilities  to  close  the  gap.  Future  resources  are  almost  always  in  tight 
supply.  Thus,  in  the  absence  of  increased  total  funding  for  DoD,  there  is  a  compelling 
need  to  identify,  as  a  source  of  funds,  things  that  are  currently  funded  but  assessed  to  be 
less  critical  in  the  future  than  capabilities  needed  to  close  or  reduce  the  gap.  That  task 
ranks,  along  with  the  sound  construction  and  analysis  of  gap-filling  capability 
alternatives,  among  the  key  tasks  the  EoA  study  team  must  address. 

This  paper  suggests  a  theoretical  framework  for  portfolio  analysis  in  the  CD 
context.  That  approach  must  be  tempered  when  it  encounters  the  real  world.  The  need 
for  EOA  study  teams,  some  of  them  cross-service  in  composition,  to  identify  through 
analysis  a  suitable  source  of  funding  from  within  portfolio-related  programs  is  a  difficult 
challenge.  But  unless  the  discipline  needed  to  do  so  is  exercised,  the  primary  benefit  of 
“portfolio  management”  will  be  lost  and  the  decision-making  process  will  revert  to  the 
historical  practice  in  which  the  requirements  community,  aided  and  abetted  by  the 
acquisition  community,  identified  effective  ways  to  reduce  risk  by  spending  more  money 
than  currently  planned,  thus  leaving  to  the  program/budget  community  the  task  of  finding 
funding  sources  for  those  initiatives.  The  result  would  be  a  less  coherent  overall  defense 
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program  than  would  be  the  case  if  each  set  of  mission  area  experts  maximized  their  own 
capabilities  within  currently  planned  spending.  After  accomplishing  optimization  within 
portfolios,  a  higher-level  risk  assessment  and  reallocation  decision  forum — such  as  the 
current  DAWG  (Deputy’s  Advisory  Working  Group)  process — could  conduct  the 
essential  cross-mission  risk  assessments  and  reallocations  prior  to  finalizing  the  next 
Future  Years  Defense  Program  (FYDP). 

Not  only  can  the  approach  described  in  this  paper  benefit  the  ongoing  four  CD 
pilot  EoAs,  it  also  has  direct  applicability  to  the  Department’s  larger  portfolio 
management  initiatives  being  pursued  under  the  QDR-directed  IR&G  process.  The 
portfolio  analyses  being  conducted  in  both  the  CD  and  IR&G  pilots  should  be  examined 
for  lessons  applicable  to  creating  a  more  complete  and  accurate  “baseline”  procedure  to 
be  used  even  more  broadly  throughout  the  Department  as  the  portfolio  management 
concept  continues  to  expand. 

Ultimately,  the  success  of  the  CD  initiative  will  be  detennined  by  the  prevalence 
of  acquisition  programs  that  provide  timely  responses  to  the  needs  of  the  future  U.S. 
combat  forces.  That  success  depends,  in  turn,  on  sound,  timely  analytical  support  to  the 
responsible  DoD  decision-makers.  The  principles  articulated  in  the  paper  are  intended  to 
further  that  end. 

Although  it  was  not  a  primary  purpose  of  this  paper  to  develop  findings  and 
recommendations,  four  recommendations  have  emerged  from  the  effort: 

•  The  two  ongoing  DoD  portfolio  management  experiments,  i.e.  within  the  CD 
and  the  IR&G  processes,  should  be  harmonized.  “Lessons  learned”  from 
those  experiments  should  be  assessed,  documented,  and  shared  between  the 
two  communities. 

•  The  need  for  better  analytical  tools  applicable  to  DoD  portfolio  management 
is  evident.  Starting  from  the  “lessons  learned”  analysis  recommended  above, 
gaps  in  current  analytical  methodologies,  tools  and  data  should  be 
documented  and  approaches  to  mitigating  those  gaps  identified.  Once  that  is 
accomplished,  an  effort  to  develop  improved  methodologies  should  be 
undertaken. 

•  The  systems  engineering  process  should  be  applied  early  in  systems 
acquisition  to  evaluate  technical  and  engineering  risk.  To  avoid  initiating 
programs  prematurely,  such  an  evaluation  should  become  part  of  the  basis  for 
a  decision  on  the  next  stage  of  development. 
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•  The  current  uncertainty  needs  to  be  resolved  concerning  the  extent  to  which 
the  CD  process  will  continue  to  be  used: 

o  Primarily  for  reaching  decisions  strictly  on  materiel  solutions,  or 

o  As  a  broader  forum  wherein  materiel  and  non-materiel  solutions  are 
considered  along  with  tradeoffs  on  equipment  quantities  and  associated 
force  structure. 
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I.  INTRODUCTION 


The  Concept  Decision  (CD)  process  was  launched  by  the  2006  Quadrennial 
Defense  Review,  which  states: 

Assessments  of  potential  solutions  should  be  informed  by  the  acquisition 
community’s  judgment  of  technological  feasibility  and  cost-per- increment  of 
capability  improvement,  and  by  the  resource  community’s  assessment  of 
affordability.  These  inputs  will  be  provided  early  in  the  decision-making  process, 
before  significant  resources  are  committed. 1 2 3 

Under  the  direction  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics;  the  Director,  Program  Analysis  and  Evaluation;  and  the  Vice  Chainnan  of  the 
Joint  Chiefs  of  Staff  (the  Tri-Chairs),  the  CD  process  is  designed  to  better  harmonize  the 
Department’s  three  major  decision-support  processes— requirements  generation, 
acquisition  management,  and  resource  allocation — for  major  acquisition  and  joint 
programs.”  The  benefits  are  expected  to  include: 

•  Greater  future  stability  of  acquisition  programs  that  have  been  subjected  to  the 
early  Concept  Decision  process 

•  Faster  development  and  fielding  of  new  military  systems 

•  More  explicit  consideration  of  potential  non-materiel  solutions  to  important 
capability  gaps,  and 

•  The  analytic  identification  of  systems  for  potential  divestment  to  serve  as 
sources  of  funding  for  added  capabilities  from  within  the  same  general 
mission  area  or  portfolio. 

The  CD  purpose  is  to  “make  an  early  decision  on  an  affordable  capability 
investment  strategy  responsive  to  joint  warfighter  needs.”  The  measure  of  success  is  the 
creation  of  sufficient  information,  in  a  portfolio  based  context,  to  make  an  early  and 
affordable  capability  investment  decision  yielding  predictable  performance.  Four  pilot 
programs  are  under  way  to  develop  an  experiential  base  from  which  implementation 
guidance  can  be  constructed. 


1  Quadrennial  Defense  Review  Report,  February  6,  2006,  p.  67. 

2  There  will  likely  be  a  similar  process  at  the  DoD  Component  level  for  non-major  acquisition. 
Component-specific  programs.  We  use  the  term  “(CD)  decision-making  body’’  in  this  paper  for  the 
Tri-Chair  or  the  Component-level  counterpart,  as  appropriate. 

3  October  30,  2006  Tri-Chair  meeting. 
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The  primary  purpose  of  this  paper  is  to  develop  an  approach  to  portfolio  analysis 
in  the  CD  process  and  to  guide,  in  a  general  way,  future  practitioners  in  undertaking  CD 
Evaluations  of  Alternatives  (EoAs).  The  basic  structure  of  the  analyses  is  to  identify 
zero-net-cost  changes  to  the  appropriate  portfolio  that  would  result  in  improved 
capabilities  to  meet  specific  challenges.  The  approach  to  portfolio  analysis  presented  in 
this  paper  are  applicable  to  both  the  tasks  of  the  Institutional  Reform  and  Governance 
(IR&G)  Capabilities  Portfolio  Managers  (CPMs)  (another,  more  far-reaching  initiative  of 
the  2006  QDR,  described  later)  and  the  tasks  of  the  CD  EoA  Study  Directors,  which  is 
the  main  topic  of  this  paper. 

Although  the  general  format  of  the  paper  is  that  of  an  annotated  briefing,  on  some 
pages  more  extensive  remarks  are  included  that  go  beyond  material  covered  in  the 
briefing  charts.  The  appendices  also  provide  supporting  material  in  considerably  more 
detail. 
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II.  BACKGROUND  AND  PORTFOLIO  ANALYSIS  IN  THE  CD 

CONTEXT 


As  reflected  in  Figure  1,  portfolio  analysis  is  part  of  the  Evaluation  of 
Alternatives,  which  replaces  the  Functional  Solutions  Analysis  (FSA)  and  the  Analysis  of 
Alternatives  (AoA)  of  the  process  that  existed  prior  to  the  CD  initiative.  (The  components 
of  those  processes  and  their  relationship  with  the  CD  process  are  discussed  in  more  detail 
in  Chapter  III,  Sections  B  and  D). 


Concept  Decision  Portfolio  Analysis 

•  CD  normally  directed  when  a  capability  gap 
may  require  a  new  acquisition  initiative 

•  The  supporting  Evaluation  of  Alternatives 
(EoA) 

-  Similar  to  an  AoA  (Analysis  of  Alternatives),  but  with 
wider  scope  and  of  shorter  duration.  It  subsumes  the 
old  FSA  (Functional  Solutions  Analysis) 

•  A  portfolio  analysis  is  part  of  the  EoA 

•  The  QDR  decision  memorandum  directed 
that  the  CD  process  be  “implemented  now” 


Figure  1.  Concept  Decision  Portfolio  Analysis 
A.  ORIGINS  OF  PORTFOLIO  MANAGEMENT 

Before  discussing  the  particular  application  of  portfolio  analysis  techniques  in  the 
CD/EoA  context,  we  will  begin  with  a  short  discussion  of  the  origins  and  current 
application  of  portfolio  management  and  analysis  outside  the  DoD  (largely  in  private 
industry). 

The  graph  in  Figure  2  illustrates  “classical”  investment  portfolio  theory,  in  which 
an  investment  mix  is  determined  to  provide  the  greatest  return  consistent  with  an 
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acceptable  level  of  risk.4  The  “efficient  frontier”  represents  a  set  of  portfolios  that 
provide  the  greatest  expected  return  for  a  fixed  level  of  risk,  or  conversely,  the  least  risk 
for  a  fixed  expected  return.5 

Although  this  concept  may  not  apply,  in  a  mathematical  sense,  to  portfolios  in  the 
Department  of  Defense,  it  has  validity  at  the  conceptual  level.  The  analog  might  be  stated 
as  follows:  for  a  fixed  level  of  capability  needed  (analogous  to  the  “expected  return”  in 
the  MPT),  DoD  should  seek  portfolio  mixes  of  capabilities  that  provide  the  desired 
aggregate  capability  at  minimum  risk.  Alternatively,  one  could  fix  the  level  of  risk  to  an 
acceptable  level  and  seek  portfolio  mixes  of  capabilities  that  would  close  the  gap  at 
minimum  cost. 

The  basic  concept  can  be  generalized  to  represent  “efficient  frontier”  choices  for 
such  variables  as  performance  versus  cost  or  risk  versus  cost. 


“Modern  Portfolio  Theory”  (MPT) 


•  MPT  (Markowitz)  states  the 
relationship  between  return  and 
risk  for  an  investment 

-  if  you  want  greater  return,  you  have 
to  accept  more  risk 

•  Problems: 

-  Can’t  predict  future  returns  or  future 
risks 

-  Variance  is  not  necessarily  a  good 
measure  of  risk 

*  Does  this  have  an  analog  in 
defense  portfolio  analysis? 

-  The  return  is  a  future  capability 

-  Risk  is  risk  of  achieving  it,  or  that  it 
will  be  sufficient 

*  In  this  construct,  if  we  fix  the 
capability  to  that  adequate  to  fill 
a  gap,  then  the  analog  is  to  seek 
to  achieve  that  capability  with 
minimum  risks 


( Variance ) 


Figure  2.  “Modern  Portfolio  Theory”  and  its  Applicability  to  Portfolio  Analysis  in  the  DoD 

Context 


B.  PORTFOLIO  MANAGEMENT  AND  ANALYSIS  IN  THE  PRIVATE 
SECTOR 

Figure  3  outlines  the  concept  of  portfolio  management  and  analysis  as  used  in 
private  industry  today.  In  some  industries,  this  is  a  routine  part  of  strategic  planning 


Harry  Markowitz  won  a  Nobel  Prize  in  economics  in  1990  for  work  that  included  this  theory. 
Risk  here  means  the  probability  that  an  unfavorable  outcome  will  occur. 
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within  each  product  line,  or  business  area.  A  recent  Government  Accountability  Office 
(GAO)  report  6  described  the  process  used  by  five  prominent  companies  in  the  private 
sector  as  follows: 

To  ensure  [achieving]  a  balanced  mix  of  executable  development  programs,  the 
successful  commercial  companies  . .  .use  a  disciplined  and  integrated  approach  to 
prioritize  market  needs  and  allocate  resources.  This  approach,  known  as 
portfolio  management,  requires  companies  to  view  each  of  their  investments 
from  an  enterprise  level  as  contributing  to  the  collective  whole,  rather  than  as 
independent  and  unrelated.  . . .  [A]  portfolio  management  approach  begins  with 
an  enterprise-level  identification  and  definition  of  market  opportunities  and  then 
prioritization  of  those  opportunities  within  resource  constraints. 

A  key  feature  of  the  private  industry  processes  described  by  GAO  is  the 
winnowing  down  of  investment  choices  as  more  information  about  the  potential  payoffs, 
risks,  and  costs  become  available  to  decision-makers,  who  must  make  tradeoffs  within 
limited  resources  to  achieve  adequate  returns  with  acceptable  risks.  Success  is  critical  to 
the  very  future  survival  of  the  company. 


Portfolio  Management  and  Analysis 
in  the  Private  Sector 

•  Portfolio  management 

-  A  mainstream  tool  of  business  management 

•  Portfolio  management  is  used  to  select  a  portfolio  of 
new  product  development  projects  to  achieve  the 
following  goals: 

-  Maximize  the  profitability  or  value  of  the  portfolio 

-  Provide  balance 

-  Control  risks 

-  Support  the  strategy  of  the  enterprise 

•  Responsibility  of  the  senior  management  team  of  an 
organization  or  business  unit 

•  Portfolio  analysis 

-  A  prerequisite  for  sound  portfolio  management 


Figure  3.  Portfolio  Management  and  Analysis  in  the  Private  Sector 

While  the  GAO  report  emphasizes  that  portfolio  management  should  be  done  at 
the  enterprise  level,  that  is  very  difficult  at  DoD,  as  it  is  in  very  large  diversified 
companies.  Although  difficult,  such  tradeoffs  must  and  will  be  made,  either  explicitly  or 


6  Best  Practices:  An  Integrated  Portfolio  Management  Approach  to  Weapon  System  Invests  Count 
Improve  DoD 's  Acquisition  Outcomes,  GAO-07-388,  March  2007. 
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implicitly,  at  the  DoD  enterprise  level.  Our  concern  in  the  CD  context,  however,  is  not 
enterprise-level  portfolio  management,7 *  but  rather  more  limited  portfolios  covering  a 
capability  area. 

Companies  invest  in  R&D  as  part  of  their  investment  portfolios  for  several 
reasons:  to  improve  products  to  maintain  competitive  posture;  to  reduce  risks  by 
providing  alternative  products  and  product  components;  and  to  achieve  break-throughs 
with  the  potential  for  great  opportunities  in  the  market  place  if  successful.  Historically, 
the  Defense  Department  has  hedged  in  similar  ways  by  having  the  Defense  Advanced 
Research  Projects  Agency  (DARPA)  and  some  other  organizations  invest  in  high-risk, 
high-potential-payoff  activities.  It  has  done  less  well  within  major  acquisition  programs 

o 

of  record,  often  short-changing  risk-reduction  initiatives,  as  well  as  failing  to  establish 
the  reserves  that  will  predictably  be  needed.9 


7  The  domain  of  the  IR&G  portfolios  mentioned  above. 

s  The  phrase  “program  of  record”  is  used  to  mean  a  DoD  activity,  or  group  of  activities,  that  has  funding 
in  the  Future  Years  Defense  Program  (FYDP). 

9  By  “predictable”  we  mean  that,  out  of  a  given  set  of  programs,  some  are  likely  to  require  management 
reserves.  It  can’t  usually  be  predicted  which  particular  program  will  need  it.  Unfortunately, 
maintaining  management  reserves  on  any  consistent  basis  appears  to  be  “a  bridge  too  far”  for  DoD. 
Such  funds  are  simply  too  attractive  as  “bill-payers”  when  short-term  budgetary  shortfalls  have  to  be 
met. 
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III.  PORTFOLIO  MANAGEMENT  AND  ANALYSIS 
IN  THE  CD  PROCESS 


The  Department’s  renewed  emphasis  on  Capabilities  Based  Planning  is  intended 
to  increase  attention  to  the  “output”  of  the  Department’s  investments,  rather  than  the 
input  “stove  pipes.”  Interest  in  managing  by  output,  “mission  areas”  or  “capability 
areas,”  has  a  long  history  in  defense  management  theory,  but  has  yet  to  be  fully 
harmonized  with  the  concomitant  need  to  structure  budgetary  inputs.  The  CD  initiative, 
with  its  use  of  capability  portfolios,  is  a  major  step  in  this  management  improvement 
process. 

A.  PORTFOLIOS 

Figure  4  defines  “portfolio”  within  the  CD  process. 

What  Does  “Portfolio”  Mean 
in  the  DoD  Context? 


*  A  portfolio  is  a  set  of  existing  programs  funded  in 
the  FYDP,  plus  additional  potential  programs  or 
initiatives,  that  provide  capabilities  within  the 
domain  under  consideration 

-  The  management  and  analysis  challenge  is  less  complex  if 
portfolios  are  mutually  exclusive 

*  If  portfolios  overlap,  subsequent  deconfliction  is  needed 

*  Alternative  combinations  of  existing  and  additional 
programs  to  be  considered  for  closing  the  identified 
gap,  each  with  particular  cost  and  risk  parameters 

*  Portfolios  should  be  analyzed  and  managed  within 
the  total  programmed  funding  before  imbalances 
across  portfolios  are  addressed 


Figure  4.  “Portfolios”  in  the  DoD  Context 
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Within  the  CD  process,  a  portfolio  is  defined  as  a  set  of  materiel  and  non-materiel 
programs  that  correspond  to  a  capability.10  The  specific  programmatic  content  of  the 
portfolio  to  be  analyzed  should  ideally  be  defined  in  the  EoA  Study  Plan,  if  not  provided 
in  the  EoA  Guidance.  If  it  is  not  provided,  an  early  priority  of  the  EoA  study  team  should 
be  to  establish  the  appropriate  capability  portfolio.  Even  if  provided,  the  portfolio  would 
need  careful  review  for  completeness  and  appropriateness  of  the  programs  included.  At 
this  early  stage  of  utilization  of  the  portfolio  management  concept  within  the  Department, 
it  is  unrealistic  to  expect  exact  identification  of  100  percent  of  the  programs  that  logically 
belong  to  such  portfolios,  but  it  is  important  that  the  study  teams  be  able  to  start  their 
work  promptly  with  the  content  of  the  vast  majority  of  the  portfolio’s  programs  clearly 
defined. 

B.  CAPABILITY  GAPS 

Figure  5  outlines  the  concept  of  “capability  gaps”  within  the  CD  context.  It  is 
essential  that  the  analyses  of  capability  gaps  begin  with  national  military  goals  and 
objectives  as  prescribed  in  the  President’s  National  Security  Strategy,  the  Secretary  of 
Defense’s  National  Defense  Strategy,  the  Chairman’s  National  Military  Strategy,  and  the 
Secretary  of  Defense’s  Guidance  for  the  Development  of  the  Force.11 


10  The  relationship  to  the  Joint  Capability  Areas  (under  development  in  a  task  being  led  by  the  Joint 
Staff/J-7  under  direction  from  the  Deputy  Secretary  of  Defense)  will  be  discussed  subsequently. 
Formerly  the  Strategic  Planning  Guidance  and  the  Contingency  Planning  Guidance 
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Capability  Gaps 

*  Capability  gaps  are  shortfalls  in  the  ability  of  the  programmed  forces  to 
meet  the  military  demands  implied  by  national  security  objectives 

*  The  Joint  Staff’s  “Joint  Capabilities  Integration  and  Development 
System”  (JCIDS)  process  identifies  key  capability  gaps 

*  The  “Capabilities  Based  Assessment”  (CBA)  is  the  analytical  process 
within  JCIDS  for  assessing  and  documenting  capability  gaps.  As  part 
of  a  CBA,  the  Functional  Needs  Analysis  (FNA): 

-  Assesses  the  full  range  of  capabilities,  materiel  and  non-materiel 

-  Identifies  redundancies  or  overlaps  in  capability  areas 

-  Describes  and  prioritizes  gaps,  assessing  impact  and  risks 

*  Gaps  can  also  be  identified  directly  by  the  Secretary’s  Portfolio 
Managers  or  by  the  Combatant  Commanders 

*  The  portfolio  analysis  should: 

-  Verify  the  capability  gap  and  its  characteristics  (when  beginning,  how  serious) 

-  Explore  options  for  cost-effectively  closing  capability  gap(s)  over  time 

-  Explore  options  for  disinvesting  in  capabilities  that  have  low  payoff 

-  Identify  risks  associated  with  those  options 


Figure  5.  Capability  Gaps 

The  Joint  Staff  is  responsible  for  translating  national  security  objectives  into 
military  requirements.  The  Joint  Staff  instituted  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  process  to  support  the  Joint  Requirements  Oversight 
Council  (JROC)  in  translating  strategic  priorities  into  military  operational  needs  or 
requirements  to  meet  those  priorities.  The  objective  of  JCIDS  is  to  support  the  JROC  in 

1 2 

“identifying,  assessing,  and  prioritizing  joint  military  capability  needs.” 

The  key  component  of  JCIDS  in  defining  capability  gaps  is  the  Capabilities  Based 
Assessment  (CBA)  process,  and  more  particularly  within  the  CBA,  the  Functional  Needs 
Analysis  (FNA).  The  FNA  assesses  the  capabilities  of  the  programmed  force  to  meet 

1  T 

military  objectives  (as  defined  by  a  preceding  Functional  Area  Analysis).  Based  on  the 

FNA,  a  Joint  Capabilities  Document  (JCD)  is  developed  to  document  the  capability  gaps. 

A  Functional  Needs  Assessment  should,  if  done  properly,  set  the  stage  for  the  portfolio 

analysis  part  of  the  CD  process.  CJCS  Manual  3 170.0 1C  defines  the  FNA  as  follows: 

The  FNA  assesses  the  capabilities  of  the  current  and  programmed  warfighting 
systems  to  meet  the  relevant  military  objectives  of  the  scenarios  chosen  in  the 
FAA  using  doctrinal  approaches.  Using  the  standards  and  evaluation  criteria 


12  CJCS  Instruction  3170. OIF,  “Joint  Capabilities  Integration  and  Development  System,”  May  2007. 

13  CJCS  Manual  3 1 70.0 1C,  “Operation  of  the  Joint  Capabilities  Integration  and  Development  System,” 
May  1,  2007.  Those  readers  not  familiar  with  JCIDS  may  wish  to  consult  these  documents,  since  it  is 
beyond  scope  of  this  report  to  provide  a  full  explanation  of  the  JCIDS  processes. 
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described  the  FAA,  the  FNA  assesses  whether  or  not  an  inability  to  achieve  a 
desired  effect  (a  capability  gap)  exists.  The  FNA  also  identifies  any  capability 
areas  that  may  have  overlaps  or  redundancies. 

The  CJCS  manual  also  states  that 

The  FNA  must  then  assess  the  impact  of  the  capability  gaps  in  terms  of  the  risk 
to  mission  (ability  to  achieve  the  objectives  of  the  scenario),  the  risk  to  force  (the 
potential  losses  due  to  the  capability  gap). ..The  FNA  should  offer  a  prioritization 
of  gaps  that  is  directly  linked  to  priorities  in  the  strategic  guidance,  but  the 
document  must  publish  sufficient  information  to  expose  how  these  priorities  were 
developed... It  must  also  completely  document  the  significant  driving  factors 
behind  the  recommended  priorities  to  give  senior  leaders  the  information  they 
need  to  make  adjustments. 

The  final  component  of  the  CBA  process  is  the  Functional  Solutions  Analysis. 
After  validation  of  the  JCD  by  the  JROC,  an  FSA  may  be  conducted  to  identify  viable 
solutions  to  the  capability  gap,  including  both  materiel  and  non-materiel  alternatives.  At 
the  completion  of  the  FSA,  a  sponsoring  DoD  Component  offers  materiel  solutions  in  an 
Initial  Capabilities  Document  (ICD).  Non-materiel  solutions  are  addressed  with  a 
doctrine,  organization,  training,  material  leadership  and  education,  personnel,  and 
facilities  (DOTMLPF)  Change  Request  (DCR). 

The  Joint  Staffs  CBA  process  has  identified  a  significant  number  of  gaps,  but 
few  if  any  redundancies,  thus  rendering  that  process  of  limited  value  in  balancing  ends 
and  means.  It  is  the  goal  of  the  CD  EoA  process  to  identity  such  balances,  at  least  at  the 
portfolio/mission  area  level.  By  combining  these  efforts,  a  large  number  of  potential 
alternative  DOTMLPF  solutions  for  closing  a  capability  gap  can  be  screened  (as  part  of 
the  FSA-like  analyses)  to  determine  a  small  group  of  the  most  promising  alternatives  to 
be  refined  in  more  detail  (as  part  of  the  AoA-like  efforts).  This  approach  is  designed  to 
supplant  the  extensive,  time  consuming  and  costly  sequential  FSA  and  AoA  analytical 
efforts  that  have  frequently  been  perfonned  in  the  past. 

C.  THE  TRADE  SPACE 

Figure  6  describes  the  concept  of  “trade  space”  as  it  relates  to  the  CD  process. 
Trades  within  the  portfolios  are  the  result  of  analytical  comparisons  of  the  (time- 
dependent)  operational  effectiveness,  suitability,  risk,  and  life-cycle  cost  of  alternatives 
that  satisfy  established  capability  needs.  The  EoA  team  will  be  asked  to  identify  sources 
within  the  portfolio  that  could  provide  the  resources  needed  to  implement  an  alternative 
with  minimal  adverse  impact.  The  decision-making  body  may  decide  that  the  identified 
sources  are  not  acceptable.  In  those  instances,  they  will  recommend  looking  for  resources 
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in  other  portfolios,  using  risk  assessment  and  allocation  techniques.  More  generally,  it  is 
the  Department’s  policy  to  require  portfolio  managers  to  demonstrate  that  the  contents  of 
their  portfolios  have  been  thoroughly  “scrubbed”  before  consideration  will  be  given  to 
tapping  other  capability  portfolios  to  obtain  needed  resources. 


What  Does  “Trade  Space”  Mean  in  the 
Context  of  Portfolio  Analysis? 

*  In  this  context,  the  “trade  space”  is  the  set  of  existing  portfolio 
programs  that  contribute  to  the  subject  capability  area  (materiel 
and  non-materiel,  funded  in  FYDP)  that  could  be  a  source  for 
divestitures  to  pay  for  a  capability  gap-filler 

-  Study  leaders  need  to  be  careful  when  valuing  programs  that  also 
contribute  to  other  capability  portfolios 

*  Decision-makers  (e.g.,  the  CD  Tri-Chair,  or  the  DAWG)  may  expand 
the  trade  space  if  acceptable  portfolio  divestitures  are  insufficient 
to  fund  needed  gap-filling  initiative(s) 

*  An  appropriate  divestiture  strategy: 

-  Analytically  identify  programs  that  are  the  least  cost-effective  in  contributing 
to  mission  capability 

•  Consider  programs  of  decreasing  priority  due  to  changes  in  the  geo¬ 
political  environment  or  threat 

-  Identify  “broken”  programs — those  no  longer  meeting  acceptable  risks  with 
regard  to  performance,  cost,  and/or  schedule 


Figure  6.  Defining  the  Trade  Space  for  Portfolio  Analysis 


D.  AoAs  AND  EoAs 

The  EoA  of  the  CD  process  is  intended  to  be  a  streamlined  analytic  methodology 
to  support  the  specific  capability  development  decision  at  hand.14  The  EoA  creates  an 
analytical  linkage  between  the  requirements  and  the  acquisition  processes,  and  when 
combined  with  the  trade  space  analysis  described  above,  it  extends  that  linkage  to  the 
Department’s  resource  allocation  process  (i.e.,  the  planning,  programming,  and  budgeting 
system).  The  EoA  combines  the  objectives  of  the  FSA  component  of  the  JCIDS,  with  the 
objectives  of  the  Analysis  of  Alternatives,  which  is  now  routinely  conducted  at  the  front 
end  of  the  acquisition  management  process  as  defined  by  Department  of  Defense 
Instruction  (DoDI)  5000.2.  Another  difference  is  that  an  EoA  should  consider  a  far 
broader  range  of  potential  solutions,  including,  in  particular,  non-materiel  solutions, 


14 


Traditional  AoAs  typically  took  two  years  or  more  to  complete,  costing  several  million  dollars.  An 
EoA,  on  the  other  hand,  should  be  completed  in  less  than  a  year,  at  considerably  lower  cost. 
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whereas  AoAs  usually  considered  only  narrowly-defined  materiel  solutions  designed  to 
justify  or  refine  a  “pre-anointed”  choice. 

According  to  the  Defense  Acquisition  Guidebook,15 

An  AoA  is  an  analytical  comparison  of  the  operational  effectiveness,  suitability, 
and  life-cycle  cost  of  alternatives  that  satisfy  established  capability  needs. 

Initially,  the  AoA  process  typically  explores  numerous  conceptual  solutions  with 
the  goal  of  identifying  the  most  promising  options  .... 

The  output  of  the  CD  process  may  be  one  or  more  formal  acquisition  programs 
that  could  serve  as  candidates  for  near-term  Milestones  A,  B  or  C,16  depending  on  how 
mature  the  program  is.  In  any  case,  the  EoA  should  serve  as  the  AoA  required  by  DoDI 
5000.2.  If  the  appropriate  milestone  for  the  program  is  Milestone  A,  then  the  CD  review 
will  normally  serve  as  the  Milestone  A  review.  For  such  programs,  an  AoA  that  updates 
the  EoA  will  likely  be  required  prior  to  Milestone  B.  Likewise,  for  a  program  going 
directly  to  Milestone  B  in  the  near  term,  an  updating  AoA  will  likely  be  required  before 
Milestone  C. 

E.  THE  FOUR  PILOTS 

Soon  after  the  start  of  the  CD  initiative  a  set  of  four  pilot  programs  were 

nominated  to  serve  as  vehicles  to  explore  and  refine  the  CD  process,  particularly  with 

regard  to  EoAs.  Figure  7  describes  the  four  pilots.  The  following  extract  from  EoA  study 

guidance  document  for  the  Joint  Light  Tactical  Vehicle  pilot  explains  the  rationale: 

The  expected  pilot  outcome  will  help  address  corporate  DoD  issues  and  provide 
potential  benefits  to  the  concept  decision  process.  Specifically,  JLTV  is  expected 
to  help  identify  how  the  Department  addresses  broad  product  line  needs  which 
cut  across  Service,  joint  force  and  the  Tactical  Wheeled  Vehicle  capability 
portfolio  lines.  The  EoA  could  also  provide  a  potential  concept  decision  benefit 
of  a  better-informed  and  earlier  investment  decision  at  Milestone  B. 


15  Available  at  the  Defense  Acquisition  University  website,  http://akss.dau. m i  1/clag/. 

16  Appendix  B,  Section  2,  summarizes  the  key  aspects  of  the  DoD  acquisition  management  process  as  it 
affects  CD. 
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Concept  Decision  Pilots 

•  JLTM  -  Joint  Light  Tactical  Mobility  (HMMWV 
replacement) 

Provides  increased  force  protection,  survivability,  mobility,  range,  operational 
availability,  and  transportability  for  all  units  &  Services.  Supports  movement  of 
organic  combat  loads,  weapons,  C4ISR  and  support  systems.  Enables  agile 
maneuver/sustainment  of  modular  forces. 

•  IAMD  -  Integrated  Air/Missile  Defense 

Seek  approaches/solutions  for  63  High-risk  gaps  in  air  and  missile  defense  in  the 
2015  timeframe.  Three  separate  EoAs  established  for:  Homeland  (Air  Force), 
Ballistic  Missiles  (Navy),  Air  and  Cruise  Missiles  (Army) 

•  JRSG  -  Joint  Rapid  Scenario  Generation 

A  set  of  enterprise  services,  to  include  distributed  database  generation  capabilities 
that  support  joint  mission-oriented  analysis,  planning,  rapid  mission  rehearsal, 
training,  experimentation,  acquisition,  and  operational  test  and  evaluation  (OT&E) 
domains. 

•  GS-R  Global  Strike  Raid 

Provide  improved  abilty  to  perform  limited-duration,  single-pass,  conventional, 
precise,  kinetic  and  non-kinetic  strikes  against  high  value,  high  payoff  targets. 


Figure  7.  The  Concept  Decision  Pilots 

Two  of  the  four  pilot  EoAs  are  completed — JLTM  and  JSRG.  The  IAMD  EoA  is 
near  completion.  The  Global  Strike  Raid  EoA  was  delayed  for  many  months  due  to 
issues  regarding  access  to  classified  programs. 

An  extensive  set  of  “lessons  learned”  was  developed  by  the  CD  Working  Group 
in  August  2007.  Although  that  compilation  is  a  useful  starting  point,  it  was  constructed 
from  suggestions  offered  by  members  of  the  group,  not  a  systematic,  comparative  review 
of  the  EoAs  by  qualified  analysts.  The  result  is,  by  and  large,  a  collection  of  unexamined 
generalities.  Thus,  there  is  still  a  need  for  a  more  rigorous  and  focused  review  of  the  four 
pilot  EoA  studies  from  an  analytical  perspective  by  a  single  entity  with  the  appropriate 
expertise. 
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IV.  AN  APPROACH  TO  PORTFOLIO  ANALYSIS  FOR  CONCEPT 

DECISIONS 


Figures  8-11  outlines  a  more  formal  approach  to  performing  portfolio  analysis  in 
the  CD  context.  Appendix  A  expresses  the  basic  steps  outlined  in  Figures  8-11  as  a 
formal  mathematical  model. 

A.  DEFINE  THE  PORTFOLIO 

Figure  8  outlines  the  steps  required  to  define  the  portfolio.  The  first  step  is  to 
identify  all  (or  at  least  most)  of  the  programs  that  provide  capabilities  in  the  portfolio  and 
which  are  funded  in  the  Future  Years  Defense  Program  (FYDP),  since  those  are  the  only 
programs  to  which  the  Department  has  officially  allocated  resources  (i.e.  the  programs  of 
record).  It  is  likely  to  be  the  case  that  many  of  the  programs  contributing  capabilities  to 
the  portfolio  will  also  provide  capabilities  to  other  portfolios.  There  are  also  likely  to  be 
programs  that  support,  directly  or  indirectly,  the  primary  programs  within  the  portfolio. 
In  each  case,  it  is  essential  to  document  where  programs  to  be  included  within  the 
portfolio  contribute  to  other  portfolios.  When  the  identified  programs  have  distinct 
Program  Elements  (PE)  in  the  FYDP,  then  it  is  easy  to  determine  the  funding  flow  for  the 
program.  However,  many  times  programs  are  rolled  up  with  other  programs  in  a  FYDP 
PE.17  In  that  case,  it  is  necessary  to  request  the  FYDP  funding  streams  from  the 
Component  in  whose  budget  the  program  resides.  Once  all  programs  have  been 
identified,  the  next  step  is  to  rank-order  them  by  the  level  of  capability  provided 
(determined,  quantitatively  if  possible,  by  analysis).  This  rank-ordering  serves  two 
purposes — it  identifies  the  major  contributors  to  the  current  capability  (the  top  programs) 
and  programs  to  be  considered  for  tradeoff  (the  bottom  of  the  list). 

Ideally,  sensitivity  analysis  will  be  conducted  on  the  major  contributors  to  the 
capability  area.  If  the  FYDP  funds  the  last  10  percent  of  the  equipment  set  for  an 
important  contributor,  those  funds  may  be  relatively  less  productive  than  similar  funding 
for  the  initial  tranche  of  a  new  contributor  whose  overall  importance,  when  fully  fielded, 
might  still  not  reach  the  top  of  the  priority  list.  Understanding  the  capability 


17  See  Appendix  D  for  more  on  this  point. 
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contributions  of  the  programs  in  the  portfolio  and  the  sensitivity  of  their  contributions  at 
various  funding  levels  will  be  of  substantial  value  in  moving  to  the  next  step. 


Approach  to  Portfolio  Analyses 

1.  Define  the  Overall  Portfolio 

•  Determine  the  set  of  programs  (materiel  and  non¬ 
materiel)  with  funding  in  the  FYDP  that  provide 
capabilities  contributing  to  the  area  wherein  the 
capability  gap(s)  has  (have)  been  identified 

-  Programs  contributing  directly 

-  Programs  contributing  indirectly  or  supporting,  but  still  important 

*  Identify  other  areas  for  which  those  programs  also 
provide  capabilities 

*  Determine  the  funding  by  fiscal  year  of  the  identified 
set  of  programs 

•  Rank-order  the  programs  according  to  priority  in 
providing  capabilities,  determined  by  analysis 


Figure  8.  Approach  to  Portfolio  Analyses — Defining  the  Portfolio 
B.  DEFINE  THE  POTENTIAL  SOLUTION  SETS 

Figure  9  continues  with  steps  to  define  a  set  of  alternatives  that  appear  to  be 
capable  of  closing  the  capability  gap.  In  doing  so,  it  is  important  to  cast  as  wide  a  net  as 
possible,  considering  solutions  ranging  from  modifications  to  existing  programs  to 
technology  initiatives  and  non-materiel  solutions  (changes  to  DOTLPF — i.e. 
DOTMLPF  without  the  “M,”  such  as  increased  force  structure  using  currently 
programmed  equipment).  The  time  dimension  is  also  an  important  consideration.  A 
capability  gap  could  occur  at  any  time — near-,  mid-  or  long-term — and  may  change  over 
time.  For  example  as  a  new  threat  weapon  system  is  projected  to  emerge  in  the  future,  it 
starts  with  an  initial  operational  capability  then  to  a  full  operational  capability.  So  the 
EoA  analyst  must  take  into  account  the  time  dimension  in  formulating  solution  sets. 
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Approach  to  Portfolio  Analyses 

2.  Define  the  Potential  Solution  Sets 

•  Define  a  set  of  alternatives  that  appear 
capable  of  filling  the  identified  gap 

-  Mods  to  existing  weapon  systems 

-  New  acquisition  program  starts 

-  DOTMLPF  changes  (non-materiel 
alternatives) 

-  Hedges  with  Science  and  Technology 
programs 


Figure  9.  Approach  to  Portfolio  Analyses — Defining  the  Potential  Solution  Sets 
C.  TRADEOFF  ANAFYSIS 

Figure  10  outlines  the  steps  for  the  tradeoff  analysis.  The  EoA  context  provides 
more  rationales  for  identifying  a  program  as  an  offset  candidate  (in  addition  to  those 
discussed  in  the  Section  III.C).  Does  the  program  have  an  impact  on  the  gaps  that 
generated  the  concerns  leading  to  the  CD?  If  not,  would  its  termination  cause  other  gaps? 
If  so,  are  those  gaps  of  as  much  concern  as  the  ones  that  are  the  basis  of  the  CD?  Put 
another  way,  where  do  we  want  to  accept  risk,  and  how  much?  Negative  answers  to  these 
questions  could  lead  to  placing  a  program  in  the  trade  space.  This  does  not  necessarily 
mean  that  the  program  will  ultimately  be  offered  up,  but  only  that  it  should  be  subject  to 
more  in-depth  analysis. 

Programs  that  no  longer  seem  to  offer  good  returns  on  DoD’s  investment  are  also 
good  trade-space  candidates.  There  could  be  a  number  of  reasons  for  putting  a  program 
in  that  category.  For  example,  the  program  may  be  encountering  so  many  difficulties 
(e.g.,  with  testing)  that  the  risk  of  successful  completion  is  quite  high.  Or,  the  program 
could  be  perfectly  healthy,  but  changing  international  situations  and  threat  evolution  may 
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18* 

call  into  doubt  whether  the  program  is  still  needed.  Even  if  such  analysis  has  been 
conducted  in  other  venues  (which  is  seldom),  it  is  very  likely  to  require  updating. 

This  analytic  identification  of  viable  trade  space  candidates  is  an  essential 
objective  of  the  portfolio  analysis  as  part  of  the  EoA. 


Approach  to  Portfolio  Analyses 

3.  Trade-off  analysis 

•  Ideally,  the  portfolio  alternatives  should: 

-  Comprise  a  mix  of  new  initiatives  and  divestitures 

-  Close  the  gap 

-  Be  net  zero  cost 

-  Not  incur  high  risk  from  termination  of  the  divested  programs 

•  An  obstacle  to  achieving  that  objective  is  the  problem  of 
finding  valid  measures  of  effectiveness  for  possibly 
diverse  sets  of  programs  providing  different  kinds  of 
capabilities  that  contribute  to  the  mission  area. 

•  In  light  of  such  difficulties,  it  is  usual  for  this  type 
analysis  to  proceed  in  two  phases: 

-  First,  determine  the  best  alternative  (most  cost-effective)  of  the 
mix  of  new  initiatives  that  close  the  gap 

-  Second,  seek  divestitures  that  offset  the  cost  of  the  best 
alternative  within  acceptable  risks 


Figure  10.  Approach  to  Portfolio  Analyses — Trade-off  Analysis 
D.  ASSESS  RISKS 

Figure  1 1  outlines  some  important  areas  of  risk  related  to  the  CD  process.  While 
risk  considerations  are  mentioned  throughout  this  paper,  this  section  focuses  on  risks  that 
should  be  addressed  formally  in  an  EoA,  to  include  ways  to  control  or  mitigate  the 
identified  risks. 


18  Or  put  more  subtly,  the  program  may  still  fill  a  need,  but  at  too  high  a  cost  in  relation  to  the  benefit  (for 
example,  the  need  can  be  met  with  more  cost-effective  alternatives,  or  a  risk  can  be  accepted  in 
meeting  the  need). 
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Approach  to  Portfolio  Analyses 

4.  Assess  Risks 

•  Uncertainty  in  degree  of  severity  and  timing  of  gap 

•  Impact  of  divestitures 

-  Near-,  mid-  and  long-term 

*  Affordability:  Will  funds  required  for  a  new  program 
“break  the  bank”  in  the  outyears  (beyond  the  FYDP), 
causing  unplanned  program  cancellations,  stretch¬ 
outs,  and  fiscal  distress? 

-  Large  uncertainty  in  initial  program  cost  estimates 

-  Unprogrammed  O&S  cost  bulges 

-  Opportunity  costs  and  risks  of  curtailment  of  other  programs 

*  Technology  and  Acquisition  risks 

-  Technology  readiness 

-  Performance,  Cost,  Schedule 


Figure  11.  Approach  to  Portfolio  Analyses — Assessing  Risks 

1.  Risks  in  Defining  the  Gap 

In  many,  if  not  most,  cases  the  capability  gap  will  be  a  projected  one,  not  a 
current  one.  When  that  is  the  case,  there  will  be  doubt  about  both  the  extent  of  the  gap 
and  the  timing  of  it.  Numerous  historical  examples  illustrate  that  future  threats  frequently 
do  not  materialize  as  projected.  This  risk  might  be  mitigated  by  taking  partial  steps  to 
increase  preparedness  to  respond  more  vigorously  if  the  projection  turned  out  to  be 
accurate — usually  by  pursuing  a  research  program  within  the  Science  and  Technology 
(S&T)  budget. 

2.  Risks  associated  with  divestitures 

In  order  to  be  credible,  it  is  imperative  that  the  EoA  provide  a  strong  rationale 
(backed  by  analysis  where  possible)  for  why  nominated  divestitures  can  be  taken.  As 
part  of  that  argument,  the  risks  associated  with  the  divestiture  must  be  explored  and 
explained  to  the  decision-making  body. 

3.  Affordability  risk 

The  approach  outlined  in  this  section  is  the  ideal.  It  assumes  that  the  solutions  that 
are  capable  of  filling  the  gap  can  be  funded  with  available  offsets  (and  implicitly  seeks 
the  least  damaging  such  offsets  that  will  accomplish  that  end — see  Appendix  A  for  an 
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explicit  formulation).  In  the  real  world,  however,  it  may  not  be  possible  to  execute  the 
ideal  plan.  If,  for  example,  the  solutions  are  too  expensive  relative  to  the  available  offsets, 
the  trade-off  analysis  will  not  converge  to  a  net-zero-cost  result.  In  that  case,  an 
alternative  formulation  might  be  considered: 

1.  Determine  the  total  resources  that  can  be  made  available  from  acceptable 
divestitures 

2.  Determine  the  (time-dependent)  extent  to  which  the  gap  can  be  closed  with  the 
funds  available 

3.  Evaluate  the  (time-dependent)  risks  of  not  fully  closing  the  capability  gap 

The  decision-makers  would  then  have  a  choice  of  nominating  a  larger  set  of 
programs  to  be  included  in  the  search  for  offsets  or  accepting  the  risk  of  not  fully  closing 
the  projected  gap.  In  consideration  of  the  risks  in  assessing  the  capability  gap,  such  a 
decision  could  reflect  a  strategy  of  “take  modest  steps  now  and  evaluate  the  situation 
again  in  the  future  to  see  if  more  is  really  going  to  be  needed.”  Cost  risks  also  give  rise  to 
affordability  issues;  approaches  to  mitigating  cost  risks  are  discussed  in  the  section  below 
on  cost  considerations.  In  some  cases  it  is  likely  that  a  major  emerging  gap  will  not  be 
able  to  be  reasonably  closed  with  acceptable  risk  by  trading  only  programs  lying  within  a 
narrowly-defined  portfolio  area.  To  the  extent  that  such  trades  will  need  to  be  made 
across  several  portfolios  it  is  likely  to  fall  outside  the  purview  of  the  CD  process  and  be 
handled  within  the  traditional  program  review  structure  in  which  decisions  are  made  by 
the  Deputy  Secretary.  This  issue  is  discussed  in  more  detail  in  Section  VI. 

4.  Technological  Risks 

For  many  CDs,  technological  risk  will  be  a  major  consideration,  and  technology 
risk  reduction  measures  should  be  at  the  top  of  the  list  of  programmatic  options  for  the 
decision-making  body.  A  full  discussion  of  technological  maturity  in  conjunction  with 
the  CD  process  is  included  as  Appendix  B.  The  rest  of  this  section  is  a  summary  of 
Appendix  B. 

To  accomplish  CD  objectives  through  one  or  more  prospective  acquisition 
programs,  technical  risk  must  be  adequately  addressed  in  order  to  achieve  program 
stability  and  predictable  outcomes.  Since  technical  risk  policy  is  written  for  the  current 
acquisition  management  process,  it  will  need  to  be  adapted  to  the  new  CD  process.  The 
best  practices  for  doing  so  are  a  function  of  the  type  of  materiel  investment  decision 
being  recommended  by  a  CD. 
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The  EoA  should  assess  the  advancement  degree  of  difficulty  and  the  technical 
suitability  for  application  for  every  area  of  technical  risk.  The  results  should  be  used  to 
eliminate  alternatives  where  the  technological  risk  is  too  high  and  to  determine  the 
appropriate  entry  point  into  the  acquisition  framework  for  the  remainder.  While  the  EoA 
process  itself  may  be  sufficient  to  detennine  that  an  alternative  is  ready  to  enter  the 
acquisition  management  framework  at  Milestone  A,  greater  technical  knowledge 
obtained  from  more  refined  technical  analysis  will  be  necessary  before  going  any  further. 
Such  analyses  should  be  used  to  validate  the  EoA  results  before  seeking  a  Milestone  B  or 
C  approval. 

Implementing  an  effective  approach  to  monitoring  technology  maturity  in  an  EoA 
will  help  shift  the  focus  of  acquisition  management  from  the  System  Development  and 
Demonstration  phase  to  the  period  prior  to  the  formally  defined  “acquisition 
management”  phase,  when  broad  options  are  still  available  and  can  be  aligned  to  close 
capability  gaps. 

5,  Acquisition  Risks 

Acquisition  risks  are  normally  characterized  in  tenns  of  the  three  inter-related 
parameters  of  cost,  schedule,  and  performance.  The  GAO  studies  cited  in  Appendix  B 
provide  strong  evidence  that  acquisition  risks  can  be  significantly  reduced  by  rigorous 
application  of  existing  technology  readiness  policy.  Programs  that  started  with 
Technology  Readiness  Level  6  (“system/subsystem  model  or  prototype  demonstration  in 
a  relevant  environment”)  averaged  “only”  18.8%  development  cost  growth,  while 
programs  that  started  with  less  mature  technologies  averaged  34.6%  development  cost 
growth. 

The  second  important  factor  in  reducing  acquisition  risks,  especially  in  cost  and 
schedule,  is  maintaining  stable  requirements.  The  CD  process  carries  the  promise  that, 
through  early  and  thorough  analysis  of  requirements,  a  greater  degree  of  stability  of 
performance  requirements  can  be  maintained  throughout  the  acquisition  program. 

E.  ANALYTICAL  TOOLS  LOR  EoAs 

From  an  analytical  perspective,  the  problem  presented  by  an  EoA  is  not  unique 
within  the  realm  of  defense  analysis.  Thus  there  are  existing  tools  that  should  be 
applicable.  This  study  did  not  conduct  a  thorough  review  of  the  available  tools  with  an 
assessment  of  their  applicability  to  EoAs.  Such  an  effort  would  be  a  major  undertaking, 
well  beyond  scope  of  the  current  study.  However,  three  methodologies  that  came  to  light 
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as  noteworthy  in  potential  applicability  to  EoAs  will  be  discussed  in  this  section.  In 
addition,  since  AoAs  (and  their  predecessor,  Cost  and  Operational  Effectiveness 
Analyses  (COEAs))  have  been  undertaken  by  DoD  for  a  number  of  years  (at  least  30), 
there  is  a  rich  knowledge  base  on  methodologies,  tools,  and  lessons  learned  from  those 
analyses.  Although  AoAs,  by  nature,  are  more  narrowly  focused  than  what  is  envisioned 
for  EoAs,  the  analytical  principles  are  very  similar.  The  Office  of  Aerospace  Studies,  Air 
Force  Materiel  Command,  at  Kirtland  Air  Force  Base  maintains  a  database  of  AoA 
studies  that  covers  all  the  Military  Services.  They  have  also  published  a  useful  guide  for 
conducting  AoAs.19 

Three  methodologies  more  specifically  tailored  for  use  in  EoAs  are  discussed 

below. 

1.  The  RAND  Corporation  Methodologies 

Paul  K.  Davis,  of  the  RAND  Corporation,  has  for  many  years  been  at  the 
forefront  of  defense  analysis  from  a  capabilities  standpoint.  In  fact,  his  seminal  work  led 
to  the  adoption  of  capabilities  based  planning  by  DoD  at  the  beginning  of  the  current 
Administration.  Davis  and  his  colleagues  at  RAND  have  developed  a  family  of  tools  of 
varying  levels  of  complexity  to  assist  capabilities  based  analyses  and  analysis  of 
portfolios,  to  include  EoAs.  Those  methods  have  been  documented  in  a  recent  RAND 
paper,  and  are  only  briefly  discussed  here.  The  methodology  comprises  two  models 
based  in  Microsoft  Excel®.  The  first  is  the  Building  Blocks  to  Composite  Options  Tool 
(BCOT).  It  accepts  inputs  of  assessments  of  portfolio  components  (“building  blocks”) 
effectiveness  and  cost,  then  displays  a  scatter  diagram  of  cost  versus  effectiveness,  in 
order  to  identify  the  “efficient  frontier”  for  the  portfolio  choices.  This  technique 
facilitates  tradeoff  analysis  because  the  less  cost-effective  components  of  the  portfolio 
will  appear  well  below  the  efficient  frontier  in  the  display.  (Caution  must  be  exercised, 
though,  because  realistically,  it  is  unlikely  that  any  one  measure  of  effectiveness  will  be 
able  to  adequately  represent  overall  capabilities.  Under  another  measure  of  effectiveness, 
the  system  in  question  may  be  near  the  efficient  frontier.) 

The  second  method  is  the  Portfolio  Analysis  Tool  (PAT),  which  displays 
composite  “stoplight  charts”  for  subsets  of  portfolio  options.  This  method  produces  a 
“spanning  set”  of  scenarios  that  test  military  capabilities.  The  distinguishing  feature  of 

19  Analysis  Handbook— A  Guide  for  Performing  Analysis  Studies:  For  Analysis  of  Alternatives  or 
Functional  Solution  Analyses,  Office  of  Aerospace  Studies,  Air  Force  Materiel  Command,  July  2004. 

20  Paul  K.  Davis,  Russell  D.  Shaver,  and  Justin  Beck,  Portfolio  Analysis  Methods  for  Assessing 
Capability  Options,  RAND,  2007. 
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PAT  is  the  ability  to  “drill  down”  to  determine  what  parameters  drive  the  color  of  the 
“stoplight”  entry.  The  drill-down  can  span  several  levels. 

While  these  methodologies  appear  useful,  the  problem  lies  in  the  difficulties  in 
determining  the  basic  cost  and  effectiveness  values  needed  as  inputs.  The  methodologies 
are  basically  ways  to  aggregate  and  display  the  results  of  lower-level  analyses  of  system 
effectiveness  and  costs.  (RAND  has,  of  course,  developed  numerous  analytical  tools  for 
assessing  system  effectiveness,  to  include  campaign-level  assessments.  But  it  is  well 
beyond  our  scope  to  discuss  them  here.) 

2.  The  Mitre  Corporation  PALMA  Tool 

Like  RAND,  The  Mitre  Corporation  has  a  rich  history  of  developing  analytical 
tools  and  performing  analytical  assessments  that  are  relevant  to  the  type  of  analysis 
required  for  EoAs.  The  Portfolio  Analysis  Machine  Tool  (PALMA®)  is  “a  computer 
program  that  helps  decision-makers  to  select  the  best  portfolio  (combination)  of 
investments  from  a  set  of  potential  investment  options.”'  The  tool  is  based  on  a  mission 
tree  structure,  wherein  a  “strategy-to-task”  tree  is  defined  at  several  levels.  Missions  are 
decomposed  into  sub-missions,  then  further  decomposed  into  subordinate  missions, 
functions  and  tasks.  At  the  lowest  level  (referred  to  as  a  “leaf  node”)  data  for  the  forces 
and  systems  that  provide  relevant  capabilities  are  defined  in  terms  of  effectiveness  and 
cost  (derived  externally).  PALMA  provides  an  assessment  of  which  investment  options 
should  be  funded  to  achieve  the  desired  goals  for  the  task.  Like  the  RAND  PAT,  the 
displays  are  in  terms  of  stoplight  charts.  The  perfonnance  at  each  node  in  the  hierarchy  is 
specified  by  a  number  from  1  to  100,  representing  value  judgments  by  subject-matter 
experts  made  at  the  leaf  nodes.  Perfonnance  at  higher  levels  are  computed  as  weighted 
averages  of  the  scores  at  the  contributing  lower  levels  based  on  heuristic  “importance” 
percentage  weights  (again  specified  by  subject-matter  experts).  The  numerical  ranges  are 
translated  into  red-yellow-green-blue  color  codes  for  display  purposes.  By  changing  the 
mix  of  investment  options,  the  performance  at  the  leaf  nodes  will  change,  and  those 
changes  will  “bubble  up”  to  changes  at  the  mid  and  top  levels.  Thus  the  user  can 
determine  a  mix  of  investment  options  that  provide  a  satisfactory  assessment  at  the 
mission  level.  To  assist  the  user,  the  tool  can  display  efficient  frontier  charts  of  cost 
versus  effectiveness  to  highlight  which  investment  choices  provide  the  most  effectiveness 
benefit  for  a  particular  investment  level.  The  least  cost-effective  systems  will  lie  far  away 


21  Richard  A.  Moynihan,  Investment  Analysis  using  the  Portfolio  Analysis  Machine  (PALMA®  Too)l,  The 
Mitre  Corporation,  July  2005. 
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from  the  efficient  frontier,  which  facilitates  their  identification  for  tradeoff  analysis.  The 
displays  facilitate  exploratory  analysis  by  allowing  the  user  to  include  or  exclude  an 
investment  option  with  a  simple  mouse  click.  The  user  may  also  specify  linkages  among 
investment  options,  e.g.,  “if  you  want  to  include  investment  option  A  you  must  also 
include  investment  option  B  (but  not  conversely).” 

PALMA  is  capable  of  including  a  large  number  of  investment  options  (“100  or 
more”)  through  the  use  of  search  methods  based  on  genetic  algorithms  or  integer 
programming.  A  version  of  the  model  is  also  available  allowing  time-phasing  (multi¬ 
year)  of  investment  options  and  budgets. 

PALMA  is  similar  to  the  RAND  methodology,  but  there  are  differences.  For 
example,  PAT  does  not  compute  a  weighted  score  for  the  top-level  assessment  from  sub¬ 
tier  assessment.  Instead,  it  simply  assigns  the  lowest  score  from  the  sub-tier  to  the  next 
level.  So  if  four  sub-tiers  are  rated  “green,  green,  yellow,  red,”  the  next  higher  tier  would 
be  rated  “red.”  (There  is  an  implicit  assumption  in  the  PALMA  method  that  the 
numerical  scopes  can  be  meaningfully  combined  in  linear  combinations — i.e.,  a  score  of 
20  is  twice  as  good  as  a  score  of  10 — whereas  it  is  not  clear  that  the  subject  matter 
experts  from  which  such  scores  originated  had  such  an  assumption  in  mind.) 

3.  The  Mitre  Corporation  Matrix  Mapping  Tool 

The  Matrix  Mapping  Tool  (MMT)  was  developed  for  OUSD(AT&L)  and  Joint 
Staff/J-8 

to  facilitate  cross  organization  coordination  and  reuse  in  support  of  capabilities 
based  planning,  analysis,  and  acquisition.  The  MMT  is  a  database  with 
supporting  software  that  documents  relationships  between  warfighting  activities, 
the  UJTL,22  systems,  ACTDs,23  roadmaps,  and  capability  areas.  It  allows  for  a 
common  set  of  reusable  data  to  support  portfolio  management  (functional, 
operational),  analysis  of  capability  gaps,  and  other  studies  where  it  is  necessary 
to  understand  the  relationships  across  the  dimensions  listed  above.24 

This  tool  should  be  of  particular  value  to  EoA  study  groups  because  of  its  ability 
to  display  a  wealth  of  the  pertinent  information  about  acquisition  programs  and  how  they 
link  to  capability  areas.  In  fact,  it  was  used  effectively  in  the  I  AMD  EoA.  The  tool 
provides  web  links  to  sites  containing  more  detailed  information.  For  example,  MMT 


22  Universal  Joint  Task  List. 

23  Advanced  Concept  Technology  Demonstrations. 

24  Judith  Dahmann,  Mitre  Corporation;  Kristen  Baldwin,  OUSD(AT&L)/Defense  Systems;  Ajay 
Choudhary,  Fary  Eiserman,  Jason  Femino,  Gary  Stutts,  Matrix  Mapping  Tool  (MMT),  Raytheon 
Virtual  Technology  Corporation  (undated). 


24 


has  a  linkage  to  the  Joint  Forces  Command’s  Joint  Capabilities  Mapping  Environment, 
which  contains  detailed  information  at  the  systems  engineering  level. 
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V.  COST  CONSIDERATIONS 


A.  COST  UNCERTAINTY 

Cost  is  always  a  key  consideration  in  any  decision  regarding  the  allocation  of 
scare  resources,  and  significant  levels  of  uncertainty  are  typically  associated  with  cost 
estimates  of  future  capabilities.  The  cost  uncertainties  are  substantially  higher,  however, 
for  capabilities  most  likely  to  be  considered  in  the  CD  process,  because  proposals  for  a 
new  capability  to  close  a  gap  projected  for  10-15  years  in  the  future  can  be  expected  to 
involve  new  technologies  applied  and  integrated  in  new  ways.  The  term  in  current  usage 
to  describe  this  process  is  “rough  order  of  magnitude”  (ROM)  costing.'  In  other  words 
there  will  be  some  large,  but  generally  unspecified,  level  of  uncertainty  in  a  cost  estimate. 
In  particular,  the  alternatives  under  consideration  in  a  CD  review  could  have  very 
different  levels  of  uncertainty  in  their  cost  estimates.  If  an  alternative  with  very  high  cost 
uncertainty  (say,  factor  of  two  or  more)  is  selected,  it  may  be  desirable  to  begin  with  a 
program  to  reduce  the  uncertainty  in  both  cost  and  perfonnance  (which  in  fact  go  hand  in 
hand). 

Cost  uncertainty  further  complicates  the  issue  of  when  resources  for  a  proposed 
new  capability  should  actually  be  set  aside  in  formal  programs  and  plans.  How  much  of 
a  “program  wedge”  (if  any)  should  be  established  in  the  FYDP  for  such  systems?  And  at 
what  point  should  future  funding  be  allocated?  To  program  funds  before  the  technologies 
for  a  new  system  are  proven  may  induce  turbulence  in  the  projected  out  year  resources 
available  for  other  programs. 

One  approach  to  this  problem  is  one  that  was  popular  during  the  1990s,  known  as 
“cost  as  an  independent  variable  (CAIV).”  '  While  the  terminology  is  somewhat  obscure 
(particularly  to  non-engineers),  the  concept  is  sound.  The  idea  is  to  decide  through  an 
analysis  of  all  claims  on  future  resources,  balanced  against  their  potential  payoffs,  how 
much  of  the  future  resources  can  be  allocated  reasonably  to  a  new  capability,  as  a 
function  of  time.  The  subsequent  development  of  the  capability  must  be  constrained  by 


25  Many  using  this  term  seem  unaware  of  its  traditional  meaning — an  order  of  magnitude  literally  means 
a  factor  of  ten. 

26  A  related  concept  is  “Design  to  Cost,”  but  that  approach  is  normally  considered  at  the  system  design 
level,  rather  than  in  the  resource  programming  environment. 

27 
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that  resource  allocation.  If  after  further  development  and  analysis,  it  is  determined  that 
the  allocated  resources  are  not  adequate  for  the  successful  fielding  of  the  preferred 
solution,  a  decision  must  be  made  to  either  reduce  the  costs  (and  presumably  also  the 
capability)  or  allocate  more  resources  (with  attendant  opportunity  costs  and  risks).  The 
discipline  imposed  by  this  approach  may  be  of  particular  value  in  CDs. 

B.  THE  PROBLEM  OF  TIME  HORIZONS 

A  potentially  critical  problem  that  is  very  likely  to  arise  in  an  EoA  is  identifying 
divestitures  that  will  free  up  funding  in  the  time  period  needed.  We  refer  to  this  problem 
as  the  time-horizon  issue,  which  is  described  in  Figure  12. 


The  Time-Horizon  Issue 

•  The  divestitures  needed  to  fund  the  outyears  of  a 
major  new  acquisition  program  may  provide  more 
near-term  offsets  than  needed  for  the  early  phases 
of  the  new  program,  unless  some  elements  are 
retained  as  a  hedge  while  awaiting  the  new  program 

•  There  are  other  issues: 

-  An  offset  adequate  within  the  FYDP  may  be  inadequate  to  fund 
the  post-FYDP  bulge  in  funding  as  the  new  program  goes  into 
full-scale  systems  development  and  procurement 

-  The  excess  savings  in  the  early  years  of  the  divestiture  will 
almost  certainly  be  scooped  up  in  program/budget  reviews  and 
applied  to  other  critical  funding  needs 


Figure  12.  Timing  of  Funding  Needs  and  Divesture  Benefits 

In  the  “classical”  case  of  a  new  system  concept,  the  early  stages  of  the  program 
will  only  require  low  levels  of  funding — e.g.,  for  technology  development  and 
maturation,  and  proof  of  concept  testing.  Assuming  success,  the  funding  needs  will 
likely  increase  over  time,  most  dramatically  after  Milestone  B.  The  need  will  likely  step 
up  again  by  a  factor  of  two  or  more  as  Milestone  C  is  passed.  This  will  likely  be  many 
years  into  the  future  for  major  acquisition  programs — in  most  cases  beyond  the  FYDP. 
The  time-phasing  of  divestitures  to  match  such  a  profile  is  challenging.  If  the  divestiture 

~7  The  CAIV  concept  as  articulated  in  the  current  5000.1  and  the  Defense  Acquisition  Guidebook  fails  to 
emphasize  this  key  point. 
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is  a  deployed  system,  once  it  is  identified  as  a  lower  priority,  it  is  subject  to  near-term 
cancellation,  which  may  be  more  than  adequate  to  cover  the  initial  funding  requirements 
of  the  new  program.  Excess  funds  would  be  quickly  commandeered  for  other  purposes, 
so  when  the  new  CD  program  reaches  Milestone  B,  the  funds  from  the  divestiture  may  no 
longer  be  available. 

Often  an  existing  program  will  provide  adequate  capability  for  the  near  tenn,  but 
a  future  threat  or  circumstance  indicates  that  at  some  point  in  the  future,  its  capabilities 
will  be  inadequate  (thus  the  gap).  A  new  program  would  then  be  needed  to  replace  the 
old  one.  It  could  take  10  or  more  years  to  develop,  produce,  and  deploy  a  new  capability, 
so  the  old  system  must  be  retained  until  the  new  one  is  deployed  in  sufficient  capacity  to 
counter  the  threat.  And  offsets  from  elimination  of  the  old  program  will  rarely  be 
sufficient  to  cover  the  cost  of  the  new  capability  for  many  years,  if  ever. 

The  time  horizon  problem  is  illustrated  in  Figures  13  and  14  below.  Figure  13 
illustrates  a  potential  future  capability  gap — a  new  threat  that  is  expected  to  be  deployed 
starting  in  2019.  By  2022,  the  threat  will  exceed  the  current  U.S.  capability,  creating  a 
growing  capability  gap.  A  new  system  must  be  developed  to  fill  that  gap  and  Figure  13 
illustrates  a  plausible  funding  profile.  The  existing  systems  must  be  retained  to  meet 
current  requirements,  but  they  might  be  reduced  in  quantity  somewhat,  since  in  this 
illustration  the  current  capability  exceeds  the  threat.  As  seen  in  Figure  14,  initial 
research,  development,  test  and  evaluation  (RDT&E)  funding  for  the  new  start  is  low  and 
can  be  offset  by  partial  divestiture  of  the  current  system.  However,  to  be  able  to  cover  the 
projected  future  gap,  funding  for  the  new  capability  will  ramp  up  for  systems 
development  and  procurement  in  four  or  five  years.  But  no  more  divestitures  can  be  taken 
to  offset  that  need  because  the  threat  still  must  be  countered.  Thus  a  substantial  funding 
gap  develops  in  the  mid-  and  far-term. 
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Illustrative  Example  of  Future  Capability  Gap 


♦ 
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threat 

— ■ — 

Existing  cap. 

New  cap.  to  fill  gap 

X 

Divestiture  of  existing 

cap. 

X 

Total  cap.  (existing  + 
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Figure  13.  Illustrative  Example  of  Providing  Resources  to  Fund  a  Future  Capability  Gap 


Illustration  of  Possible  Funding  Gap 


Figure  14.  Illustration  of  Possible  Funding  Gap 

This  example  is  simplistic,  of  course,  but  not  unrealistic.  In  reality,  there  will 
likely  be  other  programs  in  the  portfolio  than  can  be  decremented  to  free  up  funding. 
However,  for  a  major  new  start,  it  is  not  realistic  to  expect  that  enough  offsets  within  a 
portfolio  can  be  found  to  fully  fund  the  new  capability  through  systems  development  and 
procurement.28  Thus,  it  can  be  expected  that  offsets  outside  the  portfolio  will  be  needed. 
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The  average  funding  required  for  systems  development  and  procurement  for  programs  requiring 
Selected  Acquisition  Reports  (SARs)  over  the  1995-2005  time  period  was  $14B. 
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It  is  important  to  identify  the  likely  need  to  make  such  choices  at  the  CD  point,  even  if 
they  are  not  required  immediately. 

When  such  choices  have  to  be  made  is  an  important  parameter  as  well.  That  is  the 
point  at  which  (1)  the  future  need  is  established  beyond  reasonable  doubt;  (2)  sufficiently 
accurate  estimates  of  the  amount  and  timing  of  future  funding  become  available;  and  (3) 
there  is  a  high  level  of  certainty  that  an  acquisition  program  can  be  executed  within 
acceptable  risks  in  technology  development,  cost,  schedule,  and  performance.  These 
conditions  will  not  necessarily  be  met  all  at  once.  Even  if  condition  (1)  is  established, 
systems  development  should  not  commence  until  technological  readiness  is 
demonstrated.  The  level  of  risks  that  is  acceptable  will  depend  on  the  criticality  of  filling 
the  capability  gap.  But  no  matter  how  critical  the  gap,  experience  has  demonstrated  that  it 
is  unwise  to  launch  systems  development  and  procurement  before  risks  are  fully 
understood.  Operational  work-arounds  or  strategy  changes  may  be  necessary  in  the 
interim. 
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VI.  RELATIONSHIP  WITH  OTHER  DEPARTMENTAL 

PROCESSES 


This  section  discusses  the  relationship  of  the  CD  process  to  other  Departmental 
processes,  both  existing  and  in  development.  These  include  the  IR&G  process  and  the 
Joint  Capability  Area  process  (both  in  development)  and  several  processes  within  the 
planning,  program,  and  budgeting  system.  This  discussion  covers  processes  at  the 
Departmental  level  only.  There  will  be  analogous  interfaces  at  the  DoD  Component  level. 

A.  INSTITUTIONAL  REFORM  AND  GOVERNANCE 

Institutional  Reform  and  Governance  (IR&G)  is  another  management  initiative 
that  was  recommended  by  the  most  recent  QDR.  That  initiative  is  much  more  far- 
reaching  in  scope  than  the  CD  initiative,  and  in  fact,  when  broadly  interpreted  it 
encompasses  the  CD  process.  The  interrelationship  with  the  CD  process  is  summarized  in 
Figure  15.  In  a  March  2007  memorandum,  the  Deputy  Secretary  of  Defense  outlined  an 
implementation  procedure  for  IR&G.  The  overall  process  is  described  in  Appendix  C, 
where  the  impact  of  the  implementation  memo  on  the  CD  process  is  discussed  in  detail. 
The  memo  establishes  six  “Actions  to  Critical  Path”  (ACPs)  for  IR&G  implementation, 
which  are  listed  in  Figure  16.  Several  of  the  ACPs  relate  to  portfolio  analysis  in  support 
of  the  CD  process.  Certainly,  one  of  the  major  thrusts  of  the  IR&G  initiative  is  make  far 
greater  use  of  portfolio  contexts  in  DoD  management.29 


29  As  this  paper  was  being  completed,  a  draft  DoD  Directive,  Subject:  Capability  Portfolio  Management, 
was  circulated. 
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IR&G  Portfolios 

•  Currently,  IR&G  established  portfolios  (with 
designated  portfolio  managers)  are  limited  to 
programs  of  high  joint  interest  but  not  specific  to 
any  particular  Service  (“  no  natural  owner”) 

-  Joint  command  and  control 

-  Joint  net-centric  operations 

-  Battlespace  awareness 

-  Joint  logistics 

•  There  is  little  overlap  between  the  IR&G  portfolios 
and  the  CD  pilot  portfolios 

-  That  can  be  expected  to  change  in  the  future,  as  the  two 
processes  mature 

•  Where  there  is  overlap,  the  IR&G  Capability  Portfolio 
Manager  should  be  a  major  participant  in  the  CD  EoA 


Figure  15.  Relationship  between  IR&G  and  CD  Portfolio  Analysis 


Actions  to  Critical  Path  (APCs) 

Defined  for  the  IR&G  Process 

•  ACP  1  -  Strategic  Direction 

•  ACP  2  -  Portfolio  Management 

•  ACP  3  -  Decision  Framework 

•  ACP  4  -  Process 

•  ACP  5  -  Strategic  Resource  Allocation 

•  ACP  6  -  Performance  Assessment  and 
Feedback 

Figure  16.  The  IR&G  Actions  to  Critical  Paths 

In  addition,  the  initiative  establishes  four  test  “Capability  Portfolio  Managers 
(CPMs)”  in  areas  “with  high  joint  value,  but  with  no  natural  owner,”  as  seen  in  Figure  15. 
One  of  the  roles  of  the  CPM  is  to  interact  with  the  CD  process  for  programs  falling 
within  the  assigned  portfolio.  The  memo  states  that  the  CPM  should  “provide 


34 


independent  input  to  the  CD  decision-making  body  on  capability  issues  as  necessary.”  No 
mention  is  made  of  the  CPM’s  role  in  the  EoA  process;  however,  as  “advocates”  of  their 
portfolios,  it  should  be  expected  that  the  CPMs  would  be  strong  players  in  the  trade¬ 
off/divestiture  aspects  of  the  CD  EoA.  Additionally,  the  portfolio  analysis  principles 
outlined  in  this  paper  are  as  applicable  to  the  IR&G  Capabilities  Portfolio  Managers  as 
they  are  to  the  CD  EoA  Study  Directors. 

Currently,  there  appear  to  be  no  major  overlaps  among  the  four  CD  pilots  and  the 
four  test  CPMs.  However,  the  number  of  CPMs  will  expand  in  the  future,  so  future 
overlap  problems  seem  inevitable.  Furthermore,  ACPs  number  3  and  4  suggest  the 
objective  of  much  more  far-reaching  use  of  portfolio  management  within  the  Department. 
If  that  transpires,  the  impact  on  the  CD  process  could  be  much  more  significant.  In  fact, 
the  logical  result  would  be  that  all  the  Department’s  programs  would  be  managed  much 
as  has  been  defined  for  the  CD  process.  While  that  is  a  worthy  goal,  there  are  significant 
obstacles  to  evolving  in  that  direction,  since  it  would  have  a  large  impact  on  the  resource 
allocation  role  of  the  Military  Departments. 

ACP  4,  entitled  “Process,”  contains  the  CD  process  as  one  of  its  sub-elements. 
The  major  difference  with  the  current  CD  process  is  that  the  IR&G  version  would  insert 
the  Under  Secretary  of  Defense  (Comptroller)  (USD(C))  as  a  major  player.  While  not 
explicitly  stated,  presumably  the  USD(C)  would  join  the  Tri-Chair  (thus  making  it  a 
“Quad-Chair”).  Although  that  idea  has  clear  advantages,  the  USD(C)  may  not  want  to 
play  a  strong  role  in  most  cases  because  of  the  long-range  nature  of  the  typical  CD 
outcome. 

Recent  developments  (see  footnotes  29  and  30)  indicate  that  Capability  Portfolio 
Management  is  being  considered  for  insertion  into  the  heart  of  the  DoD  Planning, 
Programming,  Budgeting  and  Execution  System.  Since  both  the  IR&G  and  the  CD 
processes  are  continuing  to  evolve,  it  is  not  possible  to  assess  the  full  interactions 
between  the  two  processes  at  this  time.  All  that  can  be  stated  is  that  there  will  be 
substantial  interactions  which  need  to  be  carefully  managed  in  the  way  ahead. 

B.  JOINT  CAPABILITY  AREAS  AND  THE  FYDP-JCA  MAPPING 

Development  of  the  Joint  Capability  Areas  (JCAs)  is  being  led  by  Joint  Staff  (J- 
7),  based  on  direction  from  the  Deputy  Secretary  of  Defense.  The  first  such  effort,  which 

30  In  a  December  18th  meeting,  the  DAWG  tentatively  approved  establishment  of  four  additional  CPMs— 
“force  support,”  “force  protection,”  “corporate  management  and  support”  and  “global  security 
engagement.” 
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began  several  years  ago,  resulted  in  a  set  of  22  Tier  1  JCAs  and  240  Tier  2  JCAs.  These 
were  widely  promulgated  in  the  Department,  and  found  some  use,  particularly  by  the 
Combatant  Commands.  There  were  criticisms,  however,  that  the  framework  was  too 
complex,  had  the  potential  for  growth  without  any  natural  limits,  focused  more  on  inputs 
than  outputs,  and  was  neither  mutually  exclusive  nor  exhaustive  (in  the  sense  that  every 
Departmental  activity  could  be  “binned”  into  one  or  more  JCA).  As  a  result,  in  a  March 
2007  memorandum,  the  Deputy  Secretary  directed  the  Joint  Staff  to  lead  an  effort  to 
redefine  the  JCAs.  Figure  17  is  a  very  brief  overview  of  the  current  revised  version, 
which  was  approved  by  the  DAWG  on  January  15,  2008.  The  question  we  address  here  is 
what  utility  the  JCAs  will  have  in  portfolio  analysis  in  the  CD  process. 

As  seen  in  the  figure,  the  approved  construct  is  for  nine  “Tier  1”  capabilities  that 
are  functionally  oriented.  The  DAWG  also  approved  35  Tier  2  capabilities,  also 
functionally  oriented.  CD  portfolios,  on  the  other  hand,  are  in  most  cases  going  to  be 
mission-oriented,  because  they  will  be  addressing  future  capability  gaps  in  the  ability  of 
DoD  forces  to  execute  required  missions.  The  mission-oriented  capabilities  are  explicitly 
represented  in  the  Tier  3  and  below  JCAs.  Further  decomposition  is  in  progress. 


Joint  Capability  Areas  (JCAs) 

*  The  purpose  of  the  JCAs  is  to  facilitate  a  common  language  to  support 
the  capabilities-based  planning  process 

*  Development  began  in  2004.  First  attempt  defined  22  Tier  1  DoD 
capabilities  and  240  Tier  2  capabilities. 

*  Redefinition  directed  by  the  Deputy  Secretary,  March  2007 

*  Current  approach  (approved  by  the  DAWG,  Jan.  2008)  is  for  nine  Tier  1 
functional  capabilities: 


-  Force  Application 

-  Protection 

-  Command  &  Control 

-  Logistics 

-  Battle  space  Awareness 

-  Force  Support 

-  Net-Centric 

-  Corporate  Management 

-  Building  Partnerships 

&  Support 

*  Because  they  are  functionally  oriented,  the  Tier  1  JCAs  will  probably 
find  limited  use  for  CD  portfolios 

•  Xxx  Tier  2  decompositions  and  yyy  Tier  3  decompositions  were  also 
approved  by  the  DAWG 


Figure  17.  Joint  Capability  Areas  and  Their  Relationship  with  CD 

As  part  of  the  QDR-recommended  shift  in  the  Department  to  capability  based 
planning,  in  May  2005  the  Deputy  Secretary  asked  the  Director,  Office  of  Program 
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Analysis  &  Evaluation  (PA&E)  to  apply  the  JCAs  “to  the  program  and  budget  data  bases 
as  appropriate  prior  to  the  FY08-18  POM  cycle,”  to  provide  a  uniform  process  to  relate 
JCAs  to  resources.  This  was  accomplished  by  establishing  a  mapping  between  the 
program  elements  (PEs)  of  the  FYDP  and  the  then  existing  22  Tier  1  JCAs  (see  Figure  18 
and  Appendix  D).  This,  by  necessity,  is  a  “many-to-many”  mapping,  since  a  particular 
PE  frequently  will  relate  to  more  than  one  JCA,  and  conversely  a  particular  JCA  will 
have  many  PEs  applicable  to  it.  While  that  characteristic  significantly  reduces  the  utility 
of  such  a  mapping  for  resource  allocation  purposes,  it,  nonetheless,  can  still  provide 
valuable  insights  into  how  the  Department  plans  to  allocate  its  resources  vis-a-vis  the 
capabilities  that  will  be  produced  by  that  allocation.  The  ultimate  goal  is  an  accurate 
depiction  of  the  relationships  between  resources  and  defense  strategy,  and  to  shed  light 
on  the  age-old  question  of  whether  the  Department  is  spending  its  money  consistent  with 
its  stated  strategic  objectives. 


The  FYDP  -  JCA  Mapping 

•  Developed  by  PA&E  and  Joint  Staff  for  the  original 
set  of  22  Tier  1  JCAs 

•  Currently  under  revision  for  the  new  redefined  JCAs 

•  A  “many-to-many”  mapping — therefore,  not  a 
mutually  exclusive,  exhaustive  decomposition  of 
DoD’s  resource  program  into  capabilities  categories 

•  Only  “forces”  program  elements  were  mapped, 
using  the  Force  and  Infrastructure  coding  of  the 
FYDP,  and  accounting  for  ~50%  of  the  FYDP  funding 


Figure  18.  The  FYDP — JCA  Mapping 

Appendix  D  describes  the  mapping  at  a  summary  level.  This  study  reviewed  the 
results  of  the  mapping  in  the  PA&E  data  warehouse  to  get  some  idea  of  its  utility  for 

T  1 

defining  CD  portfolios.  The  conclusion  reached  was  that  the  mapping  developed  for  the 
original  JCAs  would  have  some  utility  in  defining  CD  portfolios  and  detennining  the 

’ 1  Access  to  FDYP  data  is  tightly  controlled,  so  we  are  unable  to  provide  much  quantitative  detail  of  the 
results  in  this  paper. 
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resources  currently  programmed  for  the  elements  of  the  portfolio.  That  conclusion  will 
likely  hold  when  the  new  structure  is  completed  and  mapped;  however,  the  utility  will  be 
largely  at  the  Tier  3  and  below  level. 

C.  RELATIONSHIP  WITH  THE  DOD  PROGRAM  PLANNING  PROCESSES 
AND  THE  ANALYTIC  AGENDA 

Figure  19  summarizes  the  relationship  between  the  CD  process  and  the  DoD 
planning  process.  The  Defense  Planning  Scenarios  comprise  a  rich  set  of  future  conflict 
situations  in  which  it  is  plausible  that  the  U.S.  would  have  a  vital  interest  leading  to 
involvement  by  U.S.  forces.  These  scenarios  range  from  major  contingencies  (large-scale 
conflicts  with  major  powers),  to  insurgencies,  peace-keeping,  and  humanitarian 
assistance  and  disaster  relief.  Each  scenario  contains  options  for  increasing  or  decreasing 
the  level  of  difficulty  driven  by  assumptions.  These  scenarios  are  approved  at  the 
secretarial  level  for  use  in  DoD  planning  and  programming.  They  should  thus  fonn  the 
starting  point  for  choosing  scenarios  for  consideration  in  a  CD  EoA  analysis.  They  can  be 
enriched  as  necessary  to  provide  any  additional  details  that  may  be  required  for 
evaluating  alternative  capabilities. 


Relationship  to  the  DoD  Program  Planning 
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Figure  19.  Relationship  between  CD  and  the  DoD  Program  Planning  Processes 

Operational  Availability  (OA)  studies,  which  address  the  adequacy  of  currently 
planned  force  capabilities  in  many  of  the  DPSs,  also  will  in  many  cases  have  a  strong 
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relationship  with  CD  EoA  analyses,  since  one  of  the  objectives  of  the  OA  studies  is  to 
identify  gaps  in  future  DoD  capabilities. 

The  Analytic  Agenda  is  an  initiative  to  improve  the  contributions  of  analysis  to 
the  DoD  planning  and  resource  allocation  processes.  Figure  20  outlines  the  relationship 
between  the  analytic  agenda  and  portfolio  analysis  in  the  CD  process.  The  common 
ground  for  these  two  processes  is  that  both  concern  assessing  the  future  military 
capabilities  of  the  Department  vis-a-vis  the  future  challenges  faced  and  the  resources 
projected  to  be  available.  The  Analytic  Agenda,  however,  takes  a  broader  perspective 
than  CD  portfolio  analysis.  Nonetheless,  since  a  key  goal  of  the  Analytic  Agenda  is  to 
maintain  a  “warm  base”  for  modeling  and  simulation  of  future  conflicts  (scenarios, 
concepts  of  operation,  data,  models),  it  is  a  highly  likely  that  Analytic  Agenda  products 
will  provide  a  valuable  starting  point  for  the  analyses  needed  in  EoAs. 


Relationship  to  the  DoD  Analytic  Agenda 


•  Programmed  portfolio  contents  and  alternatives  should 
be  assessed  against  a  range  of  potential  operational 
demands  for  capabilities 

•  Analytic  Agenda  scenarios  and  data  (DPSs,  ConOps, 
threat  and  friendly  force  deployments,  analytic 
baseline  studies)  are  designed  for  this  purpose 

-  SECDEF-  and  CJCS-approved  descriptions  of  future  operational 
demands 

•  Portfolio  analysis  guidance  documents  or  TORs  should 
specify  the  DPSs  to  be  used  and  emphasized 

•  Operational  Availability  (OA)  study  series  should: 

-  Shape  EoA  perspectives  on  capability  gaps  and  excesses 

-  Incorporate  outcomes  of  decisions  in  future  studies 

Portfolio  analysis  should  be  conducted  with 
cognizance  of  the  Analytic  Agenda  framework 


Figure  20.  Relationship  between  CD  and  the  DoD  Analytic  Agenda 
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VII.  CONCLUDING  REMARKS,  FINDINGS  AND 
RECOMMENDATIONS 


This  paper  has  explored  a  number  of  important  considerations  in  conducting 
Concept  Decision  Evaluation  of  Alternatives,  particularly  with  regard  to  the  key  process 
of  defining  and  analyzing  the  capability  portfolio  relevant  to  the  mission  area.  The 
raison  d’etre  of  a  CD  review  is  to  address  a  critical  capability  gap  that  has  emerged 
through  other  processes-  and  to  do  so  early  enough  so  that  the  resulting  decision  can  be 
integrated  into  the  Department’s  long  range  budget  plans  without  undue  disruption.  The 
normal  expectation  is  that  currently  unplanned  resources  will  be  required  to  provide 
capabilities  to  close  the  gap.  Future  resources  are  almost  always  in  tight  supply.  Thus, 
in  the  absence  of  increased  total  funding  for  DoD,  there  is  a  compelling  need  to  identify, 
as  a  source  of  funds,  things  that  are  currently  funded  but  that  are  assessed  to  be  less 
critical  in  the  future  than  capabilities  needed  to  close  or  reduce  the  gap.  That  task  ranks, 
along  with  the  sound  construction  and  analysis  of  gap-filling  capability  alternatives, 
among  the  key  tasks  the  EoA  study  team  must  address. 

This  paper  suggests  a  theoretical  framework  for  portfolio  analysis  in  the  CD 
context.  That  approach  must  be  tempered  when  it  encounters  the  real  world.  The  need 
for  EoA  study  teams,  some  of  them  cross-service  in  composition,  to  identify  through 
analysis  a  suitable  source  of  funding  from  within  portfolio-related  programs  is  a  difficult 
challenge.  But  unless  the  discipline  needed  to  do  so  is  exercised,  the  primary  benefit  of 
“portfolio  management”  will  be  lost,  and  the  decision-making  process  will  revert  to  the 
historical  practice  in  which  the  requirements  community,  aided  and  abetted  by  the 
acquisition  community,  identified  effective  ways  to  reduce  risk  by  spending  more  money 
than  currently  planned,  thus  leaving  to  the  program/budget  community  the  task  of  finding 
funding  sources  for  those  initiatives.  The  result  would  be  a  less  coherent  overall  defense 
program  than  would  be  the  case  if  each  set  of  mission  area  experts  maximized  their  own 
capabilities  within  currently  planned  spending.  After  accomplishing  optimization  within 
portfolios,  a  higher-level  risk  assessment  and  reallocation  decision  forum — such  as  the 


,2  It  is  conceivable  of  course  that  the  gap  problem  can  be  solved  by  reconfiguration  of  existing 
capabilities  or  other  measures  not  requiring  reallocation  of  resources.  Such  “trivial  solutions,’ 
however,  will  probably  have  been  found  before  the  need  for  a  CD  is  determined. 
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current  DAWG  process — could  conduct  the  essential  cross-mission  risk  assessments  and 
reallocations  prior  to  finalizing  the  next  FYDP. 

Not  only  can  the  approach  described  in  this  paper  benefit  the  ongoing  four  CD 
pilot  EoAs,  it  also  has  direct  applicability  to  the  Department’s  larger  portfolio 
management  initiatives  being  pursued  under  the  QDR-directed  IR&G  process.  The 
portfolio  analyses  being  conducted  in  both  the  CD  and  IR&G  pilots  should  be  examined 
for  lessons  applicable  to  creating  a  more  complete  and  accurate  “baseline”  procedure  to 
be  used  even  more  broadly  throughout  the  Department  as  the  portfolio  management 
concept  continues  to  expand. 

Ultimately  the  success  of  the  CD  initiative  will  be  determined  by  the  prevalence 
of  “capability”  acquisition  programs  that  provide  timely  response  to  the  needs  of  the 
future  U.S.  combat  forces.  That  success  depends  in  turn  on  sound,  timely  analytical 
support  to  the  responsible  DoD  decision-making  body.  The  principles  articulated  in  the 
paper  are  intended  to  further  that  end. 

Although  it  was  not  a  primary  purpose  of  this  paper  to  develop  findings  and 
recommendations,  four  recommendations  have  emerged  from  the  effort: 

•  The  two  ongoing  DoD  portfolio  management  experiments,  i.e.  within  the  CD 
and  the  IR&G  processes,  should  be  harmonized.  “Lessons  learned”  from 
those  experiments  should  be  assessed,  documented,  and  shared  between  the 
two  communities. 

•  The  need  for  better  analytical  tools  applicable  to  DoD  portfolio  management 
is  evident.  Starting  from  the  “lessons  learned”  analysis  recommended  above, 
gaps  in  current  analytical  methodologies,  tools  and  data  should  be 
documented  and  approaches  to  mitigating  those  gaps  identified.  Once  that  is 
accomplished,  an  effort  to  develop  improved  methodologies  should  be 
undertaken. 

•  The  systems  engineering  process  should  be  applied  early  in  systems 
acquisition  to  evaluate  technical  and  engineering  risk.  To  avoid  initiating 
programs  prematurely,  such  an  evaluation  should  become  part  of  the  basis  for 
a  decision  on  the  next  stage  of  development. 

•  The  current  uncertainty  needs  to  be  resolved  concerning  the  extent  to  which 
the  CD  process  will  continue  to  be  used: 

o  Primarily  for  reaching  decisions  strictly  on  materiel  solutions,  or 

o  As  a  broader  forum  wherein  materiel  and  non-materiel  solutions  are 
considered  along  with  tradeoffs  on  equipment  quantities  and  associated 
force  structure. 


42 


Appendix  A 

FORMAL  MODEL  OF  PORTFOLIO  ANALYSIS 


The  charts  below  present  a  formal  mathematical  model  corresponding  to  the 
descriptive  model  presented  in  Section  IV  of  the  main  paper. 


1.  Define  the  Overall  Portfolio 

•  Determine  the  set  of  programs  with  funding  in  the  FYDP  that 
provide  capabilities  contributing  to  the  area  wherein  the 
capability  gap(s)  has(have)  been  identified 

-  Programs  contributing  directly 

-  Programs  contributing  indirectly  or  supporting 

-  Material  and  Non-material  programs 

•  Let  P  =  (py:  j=  1,  N}  denote  that  set  of  programs 

*  Identify  other  areas  for  which  those  programs  also  provide 
capabilities 

*  Determine  the  funding  by  fiscal  year  of  the  identified  set  of 
programs 


Figure  A-1.  Defining  the  Overall  Portfolio 
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2.  Define  the  Solution  Sets 

•  Define  a  set  of  alternatives  that  may  be 
capable  of  filling  the  identified  gap 

-  Mods  to  existing  weapon  systems 

-  New  acquisition  system  starts 

-  DOT_LPF  changes 

•  Alternatives  will  typically  be  constructed  as 
combinations  of  the  above: 

Aj  =  {xik,  k  =  each  x/f(is  a  program  or  initiative  providing  or 

supporting  a  capability  contributing  to  closing  the  gap 


Figure  A-2.  Defining  the  Solution  Sets 


3.  Trade-Off  analysis 

Let  C(Aj)  be  the  cost  of  alternative  A , 

Let  C(pj)  be  the  cost  of  existing  program  py 

Let  {pfj  e  R}  be  a  subset  of  P  consisting  of  the  programs  to  be 
retained  (R  is  an  index  set  for  the  programs  retained) 

Thus,  {pjij  &  R)  is  the  set  of  programs  to  be  divested 

Let  E(S )  be  the  effectiveness  of  a  set  S  of  programs  (a  member  of 
S  is  either  an  A,  or  a  p^) 

Analytical  objective: 

Solve  for  A ,  and  R  such  that 

E(  Aj  u  {prj  eR} )  closes  the  capability  gap, 

and  the  set  of  programs  {pjij  <?R}  can  be  divested  with  (collective) 
acceptable  risks,  while  minimizing: 

C(Ai)-'LC(pi:j*R) 


Figure  A-3.  Making  the  Trade-off  Analysis 
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4.  Alternative  Formulation 

This  formulation  might  apply  when  there  is  no  solution 
under  the  first  formulation 


Analytical  objective: 

Solve  for  A,  and  R  to  minimize 

E(to  close  gap)  -  E(  A,  u  { pfj  eR} ), 

where  the  set  of  programs  { Pjij  <?R}  can  be  divested  with 
(collective)  acceptable  risks,  and: 

c(Aj- icfty;  *Rj=o 


Figure  A-4.  An  Alternative  Formulation  of  the  Optimization  Process 
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Appendix  B 

ASSESSING  TECHNICAL  AND  ENGINEERING  RISKS  IN 
THE  CONCEPT  DECISION  PROCESS 


1.  Introduction 

This  appendix  is  organized  as  follows: 

•  Section  2  describes  how  the  new  Concept  Decision  (CD)  process  will  interact 
with  acquisition  management; 

•  Section  3  summarizes  policy  on  technical  and  engineering  risk; 

•  Section  4  provides  data  to  support  the  importance  of  the  technology  maturity 
component  of  technical  risk  to  acquisition; 

•  Section  5  identifies  some  best  practices  for  risk  assessment  in  the  Evaluation 
of  Alternatives — the  analytical  basis  for  the  CD; 

•  Section  6  is  a  summary. 

2.  The  Concept  Decision  Process  and  Acquisition  Management 

The  Concept  Decision  that  follows  an  Evaluations  of  Alternatives  (EoA)  may 
initiate  one  or  more  materiel  investments  essential  to  a  particular  materiel  alternative,  or 
to  different  competing  materiel  solutions.  After  the  CD  review,  there  are  three  possible 
entry  points  into  the  acquisition  management  framework,  based  largely  on  the  technical 
and  engineering  risk: 

•  Milestone  A.  When  the  prospects  for  risk  reduction  in  a  short  period  of  time 
appear  promising,  entry  into  the  Technology  Development  phase  of  the 
acquisition  process  may  be  approved  to  provide  the  time  and  resources  to 
reduce  risk  sufficiently  for  system  development.1 

•  Milestone  B.  When  all  of  the  critical  technologies  have  been  demonstrated  in 
a  relevant  environment  and  other  technical  risks  for  system  integration  appear 
manageable,  entry  into  the  System  Development  and  Demonstration  phase 
may  be  planned,  contingent  upon  completion  of  the  remainder  of  the  statutory 
and  regulatory  requirements  for  Milestone  B.  Approval  of  Milestone  B 
constitutes  the  formal  start  of  all  DoD  acquisition  programs  (except  for  naval 
ships)  for  Congressional  reporting  purposes  and  requires  that  the  next  Future 

1  Such  approval  usually  encompasses  funding  just  the  effort  needed  to  achieve  the  Milestone  B  criteria. 
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Years  Defense  Program  (FYDP)  reflect  the  future  years  funding  that  had  been 
approved  by  the  Defense  Acquisition  Executive. 

•  Milestone  C.  If  the  design  is  complete  and  production  risks  appear  low,  then 
Low  Rate  Initial  Production  may  be  planned  contingent  upon  completion  of 
the  remainder  of  the  statutory  and  regulatory  requirements  for  Milestone  C. 

This  appendix  focuses  on  the  evaluation  of  technical  risk  in  the  materiel  solution 
space.  For  potential  new  materiel  solutions,  an  important  underlying  principle  of  the  CD 
process  is  to  invest  sufficiently  in  early  risk  reduction  efforts  to  meet  the  formal  criteria 
for  acquisition  program  initiation  and  in  doing  so  limit  the  risk  of  subsequent  expensive 
delays  and  cost  growth.  That  principle  implies  that  the  analysis  perfonned  during  the 
EoA  should  assess  technical  and  engineering  risks  in  the  alternatives  under  consideration 
and  then  determine,  based  on  that  assessment,  the  extent  to  which: 

•  Additional  investments  should  be  made  in  the  alternative  at  this  time  because 
of  the  extent  of  the  cost  and/or  risks,  or  the  availability  of  other  alternatives,  or 
the  acceptability  of  the  “status  quo;” 

•  Specific  S&T  investments  should  occur  before  a  decision  can  be  responsibly 
made  on  whether  the  alternative  should  enter  the  acquisition  management 
process;  or 

•  An  alternative  is  mature  enough  to  support  entry  into  the  fonnal  acquisition 
management  process.  This  includes  technical  maturity,  as  well  as  readiness 
within  the  broader  infrastructure  to  support  design,  test  and  evaluation, 
producibility,  and  to  make  informed  judgments  about  supportability.  These 
features  are  discussed  in  more  detail  in  Section  4. 

The  Concept  Decision  that  follows  an  EoA  may  lead  to  an  Acquisition  Decision 
Memorandum  that  initiates  one  or  more  materiel  investments  essential  to  a  particular 
materiel  alternative,  or  to  different  competing  materiel  solutions.  When  the  risks  are  too 
high  to  enter  the  formal  acquisition  management  process  as  outlined  in  Section  2, 
materiel  investment  may  be  strictly  in  Science  and  Technology  (S&T). 

•  S&T  only.  If  the  alternative  holds  a  great  deal  of  promise  for  closing  a 
capability  gap,  but  the  risk  of  maturing  its  enabling  technologies  in  a  short 
period  of  time  is  high,  then  an  additional  S&T  investment  outside  of  the 
formal  acquisition  management  process  may  be  a  selected  course  of  action. 

3.  Technical  and  Engineering  Risk  Policy 

Several  paragraphs  in  DoDI  5000.2  provide  policy  on  assessing  technical  risk: 
“The  management  and  mitigation  of  technology  risk,  which  allows  less  costly  and  less 
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time-consuming  systems  development,  is  a  crucial  part  of  overall  program  management 
and  is  especially  relevant  to  meeting  cost  and  schedule  goals.”' 

•  As  part  of  the  requirements  generation  process,  it  states:  “The  capability  needs 
and  acquisition  management  systems  shall  use  Joint  Concepts,  integrated 
architectures,  and  an  analysis  of  doctrine,  organization,  training,  materiel, 
leadership  and  education,  personnel,  and  facilities  (DOTMLPF)  in  an 
integrated,  collaborative  process  to  define  desired  capabilities  to  guide  the 
development  of  affordable  systems.  ...  Representatives  from  multiple  DoD 
communities  shall  assist  in  formulating  broad,  time-phased,  operational  goals, 
and  describing  requisite  capabilities  in  the  Initial  Capabilities  Document 
(ICD).  They  shall  examine  multiple  concepts  and  materiel  approaches  to 
optimize  the  way  the  Department  of  Defense  provides  these  capabilities.  The 
examination  shall  include  robust  analyses  that  consider  affordability, 
technology  maturity,  and  responsiveness.” 

•  As  part  of  Concept  Refinement,  it  states:  “The  ICD  and  the  AoA  plan  shall 
guide  Concept  Refinement.  The  focus  of  the  AoA  is  to  refine  the  selected 
concept  documented  in  the  approved  ICD.  The  AoA  shall  assess  the  critical 
technologies  associated  with  these  concepts,  including  technology  maturity, 
technical  risk,  and,  if  necessary,  technology  maturation  and  demonstration 
needs.”4 

This  policy  will  remain  applicable  to  the  EoA  in  the  CD  process  when  the  broader 
Joint  Capabilities  Document  (JCD)  precedes  narrower  ICDs  and  the  Analysis  of 
Alternatives  (AoA)  replaces  the  EoA.  The  need  for  robust  analyses  ...  of  technology 
maturity  in  the  requirements  generation  process  applies  to  the  Functional  Solutions 
Analysis  (FSA)-like  part  of  the  EoA.  The  assessing  of...  technical  risk  ...  as  part  of  the 
Concept  Refinement  Phase  applies  to  the  AoA-like  elements  of  the  EoA. 

4.  Relationship  Between  Immature  Technologies  and  Cost/Schedule  Growth 

Formally  initiating  acquisition  programs  by  contracting  for  System  Development 
and  Demonstration  that  relies  (improperly)  on  immature  technologies  frequently  leads  to 
cost  growth  and  schedule  slippage. 

Technology  maturity  is  usually  measured  by  technology  readiness  levels  (TRLs) 
for  a  Technology  Readiness  Assessment  (TRA)  conducted  at  a  Milestone  decision.5 


2  DoDI  5000.2,  paragraph  3. 7. 2.2. 

3  DoDI  5000.2,  paragraph  3.4.1. 

4  DoDI  5000.2,  paragraph  3.5.3. 

5  The  use  of  TRLs  originated  with  the  National  Aeronautics  and  Space  Administration  (NASA)  in  the 
1980s. 
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TRLs  indicate  what  has  been  accomplished  in  the  development  of  a  technology  across 
various  levels  of  utility  to  the  user — ranging  from  theory,  to  laboratory  demonstration,  to 
prototype  demonstration,  to  proof  in  the  field.  TRLs  do  not  measure  whether  the 
technology  or  the  design  is  right  for  the  job,  the  difficulty  of  system  integration,  or  how 
easy  it  will  be  to  reach  the  next  level.  TRL  6,  which  determines  that  a  system/subsystem 
model  or  prototype  has  been  demonstrated  in  a  relevant  environment,  is  the  DoD- 
established  minimum  maturity  criterion  for  formal  program  initiation  and  entry  into  full 
System  Development  and  Demonstration.  See  Figure  B-l  for  hardware  and  software 
TRL  definitions. 


Hardware 


Software 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 


Basic  principles  observed  and  reported 
Technology  concept  and/or  application 
formulated 

Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept 

Component  and/or  breadboard 
validation  in  a  laboratory  environment 
Component  and/or  breadboard 
validation  in  a  relevant  environment 
System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 
System  prototype  demonstration  in  an 
operational  environment  r- 


Actual  system  completed  and  qualified 
through  test  and  demonstration 


Actual  system  proven  through 
successful  mission  operations 


1.  Basic  principles  observed  and  reported. 

2.  Technology  concept  and/or  application  formulated. 

3.  Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept 

4.  Module  and/or  subsystem  validation  in  a 
laboratory  environment,  i.e.  software  prototype 
development  environment 

5.  Module  and/or  subsystem  validation  in  a  relevant 
environment 

6.  Module  and/or  subsystem  validation  in  a  relevant 
end-to-end  environment 

7.  System  prototype  demonstration  in  an  operational 
high  fidelity  environment 

8.  Actual  system  completed  and  mission  qualified 
through  test  and  demonstration  in  an  operational 
environment 

9.  Actual  system  proven  through  successful  mission 
proven  operational  capabilities 


Figure  B-1.  Definition  of  Technology  Readiness  Levels 

The  Government  Accountability  Office  (GAO)  has  attempted  to  quantify  the 
effects  of  immature  technologies  on  acquisition  programs.6  According  to  a  2005  GAO 
review  of  54  DoD  programs:7 

•  Programs  that  started  with  the  preferred  level  of  technology  maturity  (TRL  7) 
averaged  9%  development  cost  growth,  a  7  month  schedule  delay,  and  a  1% 
acquisition  unit  cost  growth;  and 


6  DoD  did  not  make  any  general  comments  on  these  GAO  reports;  only  technical  comments  were 
provided.  We  are  not  aware  of  any  recent  systematic  DoD  efforts  to  develop  similar  quantitative 
estimates  of  the  impact  of  technology  immaturity. 

7  Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-05-301,  March  2005. 
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•  Programs  whose  technologies  did  not  meet  that  criterion  averaged  41% 
development  cost  growth,  a  13  month  schedule  delay,  and  a  21%  acquisition 
unit  cost  growth. 

The  2006  version  of  the  same  report  had  analogous  findings  based  on  52 
programs:8 

•  Programs  that  started  with  technologies  at  the  TRL  7  level  averaged  4.8% 
development  cost  growth  and  a  1%  acquisition  unit  cost  growth 

•  Programs  that  used  less  mature  technologies  averaged  34.9%  development 
cost  growth  and  a  27%  acquisition  unit  cost  growth 

The  2006  report  also  provided  some  results  based  on  a  TRL  6  minimum  criterion 
for  mature  technology.  While  the  differences  are  not  as  dramatic,  the  effect  is  similar. 

•  Programs  that  started  with  such  technology  maturity  averaged  18.8% 
development  cost  growth;  and 

•  Programs  that  started  with  less  mature  technologies  averaged  34.6% 
development  cost  growth. 

The  GAO  analyses  did  not  examine  the  extent  to  which  factors  other  than 
technology  immaturity,  such  as  inaccurate  cost  estimation  or  requirements  creep,  led  to 
the  observed  cost  and  schedule  growth.  Consequently,  causality  cannot  be  rigorously 
proven.  However,  the  linkage  between  technology  immaturity  and  these  other  factors  is 
so  strong,  it  is  reasonable  to  conclude  that  technology  maturity  contributes  in  a  major 
way  to  program  stability  and  predictable  outcomes. 

5.  Best  Practices  for  Considering  Technical  and  Engineering  Risk  in  an  EoA 

We  have  seen  strong  evidence  that  technical  risk  contributes  to  program 
instability.  However,  only  knowing  about  the  technical  risk  is  not  a  sufficient  basis  for 
making  a  materiel  investment  decision.  Further  analysis  of  this  risk  should  be  conducted 
to  support  an  assessment  of  the  degree  of  difficulty  of  advancement9  of  the  alternative 
along  with  the  technical  suitability  for  the  application.  This  is  the  information  most 
useful  to  decision  makers  in  deciding  whether  to  invest,  and  if  so,  the  nature  of  the 
investment. 


Defense  Acquisitions:  Assessments  of  Selected  Major  Weapon  Programs,  GAO-06-391,  March  2006. 
9  Ideas  presented  in  this  section  are  adapted  from  Systematic  Assessment  of  the  Program/Project 
Impacts  of  Technological  Advancement  and  Insertion;  James  W.  Bilbro;  George  C.  Marshall  Space 
Flight  Center;  December,  2006. 
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Conceptually,  there  is  a  third  factor  that  may  be  considered  in  such  an 
assessment — the  overall  maturity  of  the  technical  solution.  Three  broad  classification 
levels  may  be  used: 

•  It  does  not  exist  (only  a  paper  concept). 

•  It  has  been  demonstrated  in  a  laboratory. 

•  It  is  commonly  available. 

Several  complicating  factors  may  decrease  the  utility  of  this  assessment  factor. 
For  example,  a  software  algorithm  may  not  exist  but  may  be  demonstrated  very  quickly. 
On  the  other  hand,  in  the  hardware  world  a  major  scientific  breakthrough  may  be  needed 
to  demonstrate  something  in  a  laboratory  environment.  Therefore,  to  simplify  the 
assessment  process,  it  should  be  sufficient  to  focus  on  the  advancement  degree  of 
difficulty  in  the  context  of  the  acquisition  management  framework.  Advancement  degree 
of  difficulty  should  be  assessed  on  the  basis  of  the  difficulty  in  achieving  the  next 
acquisition  milestone  given  the  anticipated  entry  point  in  the  process. 

Both  the  degree  of  difficulty  of  advancement  along  with  the  technical  suitability 
for  the  application  should  be  assessed  on  the  basis  of  the  status  of  the  alternative  in  five 
areas  of  technical  risk: 

•  Technology  maturity  [note  that  examination  of  technology  maturity  during  an 
EOA  is  not  a  formal  Technology  Readiness  Assessment],1 

•  Test  and  evaluation  readiness, 

•  Design  and  integration  readiness, 

•  Manufacturing  readiness,  and 

•  Supportability  readiness. 

To  make  such  an  assessment,  certain  information  must  be  detennined  using  a 
systems  engineering  process  where  requirements  analysis,  functional  analysis,  and 
conceptual  design  take  place  iteratively  and  recursively.  To  complete  this  analytical 
systems  engineering  process,  it  will  be  incumbent  on  the  EoA  study  team  to  make  (and 

10  TRAs  are  required  at  Milestone  B  and  C  and  at  program  initiation,  typically  Milestone  A,  only  for 
ships.  A  TRA  is  a  systematic,  metrics-based  process  and  accompanying  report  that  assesses  the 
maturity  of  certain  technologies  [called  Critical  Technology  Elements  (CTEs)]  used  in  new  acquisition 
programs.  Technology  Readiness  Levels  (TRLs)  are  used  to  describe  the  maturity  of  CTEs.  The  use 
of  TRLs  is  also  not  appropriate  for  the  assessment  of  technology  maturity  during  an  EoA.  During  an 
EoA  associated  with  an  early  milestone  there  is  no  program  of  record.  So,  for  example,  it  is  not 
possible  to  demonstrate  a  component  or  sub-system  in  a  relevant  environment  because  (1)  the  system 
has  not  been  fully  defined;  (2)  the  relevant  environment  may  not  have  been  completely  defined;  and 
(3)  there  is  no  draft  Capability  Development  Document  on  which  to  base  technical  performance  goals. 
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document)  assumptions  about  the  unknowns  based  on  the  Functional  Needs  Analysis 
(FNA),  the  JCD,  and  the  scenarios  being  used  to  evaluate  alternatives.  This  sets  the 
conditions  for  the  CD  by  defining  what  is  needed.  It  does  not  specify  how  things  should 
be  done.  Elements  include: 

•  Determining  the  capabilities  to  be  provided  on  the  basis  of  the  FNA. 

•  Postulating  the  nature  of  the  materiel  solution  (the  type  of  system  it  will  be 
and  how  it  will  be  used  to  accomplish  its  mission)  under  consideration  to 
close  the  capability  gap. 

•  Making  an  initial  determination  of  the  functions  necessary  to  meet  the 
required  performance  characteristics  to  provide  the  desired  capability  on  the 
basis  of  measures  of  effectiveness  articulated  in  the  JCD. 

•  Estimating  plausible  system  operational  requirements,  operational  effec¬ 
tiveness/utility,  and  cost  based  on  assumptions  about  a  system  concept  and 
key  characteristics. 

These  assumptions  should  not  be  made  by  the  technical  community  in  a  vacuum. 
There  must  be  strong  communication  with  the  user  community.  In  that  way,  the 
functional  analysis  ultimately  forms  a  bridge  between  requirements  and  system  design  or 
systems  architecture,  by  considering  alternative  means  of  performing  the  functions  and 
by  trading  among  scarce  resource  parameters  (such  as  cost,  weight,  power,  and  space). 
Each  element  influences  and  is  influenced  by  the  others  as  tradeoffs  are  made  to  discover 
the  best  solution  on  an  iterative  basis. 

The  information  generated  by  the  systems  engineering  process  can  be  used  to 
make  a  preliminary  determination  of  enabling  technologies  for  the  alternative  under 
consideration  by  systematically  considering  all  elements  of  the  work  breakdown  structure 
or  systems  architecture.  It  is  not  necessary  to  evaluate  risks  associated  with  all 
technologies.  The  assessments  should  be  applicable  only  to  those  technologies  that  pose 
the  greatest  risk  to  the  success  of  a  potential  materiel  solution.  Therefore  technologies 
that  affect  cost  in  a  significant  way  should  be  considered  as  well  as  those  technologies 
essential  for  mission  performance.  If  the  technology  has  been  used  before  in  a  similar 
environment,  risks  are  likely  to  be  much  better  understood.  Therefore  whether  a 
technology  is  “new  or  novel”  is  another  important  criterion. 
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Critical  enabling  technologies  are  those  that  have  the  highest  risk  associated  with 
them.  Such  risk  is  determined  by  two  criteria.11  The  first  criterion  is  criticality  to  closing 
the  capability  gap.  At  least  one  of  the  following  questions  must  be  answered  “yes.” 

•  Does  the  technology  enable  an  improvement  over  current  performance  levels 
necessary  to  achieve  a  predicted  reduction  to  the  capability  gap? 

•  Does  the  technology  have  a  significant  impact  on  when  the  capability  gap  will 
be  reduced? 

•  Does  the  technology  have  a  significant  impact  on  the  extent  of  the  offsets  used 
to  resource  the  life  cycle  costs  of  the  materiel  solution? 

The  second  criterion  is  “new  or  novel.”  At  least  one  of  the  following  questions  must  be 
answered  “yes.” 

•  Is  the  technology  new  or  novel? 

•  Is  the  technology  modified? 

•  Has  the  technology  been  repackaged  such  that  a  new  relevant  environment  is 
realized? 

•  Is  the  technology  expected  to  operate  in  an  environment  and/or  achieve  a 
performance  beyond  its  original  design  intention  or  demonstrated  capability? 

In  general,  greater  knowledge  implies  lower  risk.  Knowledge  about  the  technical 
solution  along  with  the  perfonnance  needs  that  the  solution  should  support  enable  a  better 
understanding  of  the  work  effort  required  to  achieve  the  next  acquisition  milestone.  The 
CD  decision-making  body  must  be  given  an  understanding  of  this  work  effort,  the 
technical  risk,  the  cost,  and  the  performance  results  to  make  a  decision  on  a  materiel 
investment. 

Assessing  the  advancement  degree  of  difficulty  implicitly  requires  the  use  of  a 
scenario  to  make  sound  decisions  about  the  future.  Peter  Schwartz  “  suggests  an 
approach  to  scenario  building.  An  important  first  step  is  tailoring  the  scenario  to 
facilitate  the  decision  to  be  made.  The  next  step  is  to  detennine  the  controllable  and 
uncontrollable  elements  of  the  relevant  environments.  The  elements  should  be  ranked 
based  on  their  importance  and  their  certainty  in  order  to  flesh  out  the  scenario  and  to 
assess  their  implications  on  the  scenario.  Finally,  key  indicators  of  either  the  potential 
success  or  failure  of  the  technical  approach  should  be  identified. 


1 1  Adapted  from  the  process  used  to  determine  critical  technology  elements  for  a  TRA. 

12  Peter  Schwartz,  The  Art  of  the  Long  View,  Currency  Doubleday,  1991. 
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The  following  four  bullets  summarize  degree  of  difficulty  of  advancement  and 
technical  suitability  considerations  in  the  five  areas  of  technical  risk  as  a  function  of 
materiel  investment  decisions  (as  introduced  in  section  2). 

•  S&T  only 

The  implication  of  an  S&T  only  investment  is  that  the  advancement  degree  of 
difficulty  is  too  high  to  enter  the  formal  acquisition  management  process  because 
the  technology  is  too  immature.  This  is  the  case  if  the  approach  for  continued 
development  is  similar  to  existing  experience  in,  at  most,  a  small  subset  of  the 
critical  areas.  In  other  words,  there  are  so  many  more  unknowns  than  knowns, 
even  the  approach  is  nebulous.  As  far  as  suitability  is  concerned,  it  is  unlikely 
that  any  determination  can  be  made. 

It  would  be  unusual  for  potential  solutions  that  are  very  immature  technically  to 
be  offered  as  an  alternative  in  the  CD  process.  Such  alternatives  would  normally 
be  eliminated  during  the  EoA  screening  process  prior  to  the  AoA-like  analysis.  A 
particular  technology  may,  however,  offer  an  especially  high  potential  for  closing 
a  capability  gap.  In  such  a  case,  even  though  there  is  no  clear  path  for  maturing 
the  technology,  the  EoA  may  highlight  that  technology  for  investment,  regardless 
of  any  existing  S&T  projects  in  the  area.  Such  an  investment  should  provide 
resources  to  advance  and  characterize  perfonnance  in  some  environment  within  a 
specified  time  frame.  The  technology  could  be  reconsidered  for  inclusion  in  a 
materiel  solution  at  a  later  date  if  the  effort  were  successful. 

No  formal  acquisition  management  requirements  must  be  fulfilled  after  the  CD 
since  these  activities  are  outside  of  the  acquisition  management  process. 

•  Milestone  A 

Feasibility  to  enter  the  acquisition  management  process  at  Milestone  A  implies 
that  both  the  advancement  degree  of  difficulty  and  the  technical  suitability  are 
moderate.  This  is  the  case  when  (1)  it  is  reasonable  to  believe  that  the  technical 
solution  is  “right  for  the  job”  and  (2)  the  approach  for  continued  development 
ranges  from  well  within  experience  levels  (a  single  development  approach  is 
adequate)  to  sufficiently  similar  to  existing  experience  to  warrant  comparison  in 
most  critical  areas  (dual  development  approaches  should  be  pursued).  Key 
elements  of  the  development  approach  vary  as  a  function  of  the  technical  risk 
areas  as  follows: 

o  For  technology  maturity,  the  development  approach  applies  to  a 
technology  maturity  plan  leading  to  critical  technologies  being 
demonstrated  in  a  relevant  environment  as  required  before  a  subsequent 
Milestone  B 
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o  For  test  and  evaluation  readiness,  the  development  approach  applies  to  test 
equipment,  test  tooling,  test  measurement  systems,  software 

o  For  design  and  integration  readiness,  the  development  approach  applies  to 
functional  requirements,  interfaces,  constraints 

o  For  manufacturing  readiness,  the  development  approach  applies  to 
materials,  machines,  tooling,  metrology,  software 

o  For  supportability  readiness,  the  development  approach  applies  to 
maintainability,  reliability,  support  equipment,  logistics  footprint 

While  a  Milestone  A  approval  does  not  guarantee  program  initiation  at  a  later 
time,  it  does  imply  such  potential.  Therefore  ,the  Concept  Decision  should  also 
establish  requirements  in  the  form  of  criteria  to  be  met  and  information  to  be 
determined,  in  order  to  warrant  consideration  for  a  Milestone  B.  Such  criteria  and 
information  should  include  the  funding,  the  schedule,  and  the  perfonnance 
characteristics  necessary  in  all  technical  and  engineering  risk  areas.  Decision 
makers  should  also  consider  these  requirements  when  assessing  whether  the 
technical  and  engineering  risk  is  sufficiently  low,  despite  the  fact  that  there  are 
uncertainties  concerning  how  development  will  be  accomplished. 

The  Department’s  typical  approach  to  “risk”  applies  to  well-defined  quantifiable 
activities.  This  approach  does  not  capture  much  of  the  “technical  risk”  at  a 
Concept  Decision  review  for  alternatives  going  to  Milestone  A,  because  that  risk 
stems  more  from  missing  information  (unknown  unknowns),  than  from 
uncertainty  about  the  outcome  of  forecast  events.  In  addition,  there  is  no  fonnal 
process  for  dealing  with  this  type  of  technical  risk.  Despite  the  lack  of  a  formal 
process,  identification  and  mitigation  of  this  “ignorance  driven”  technical  risk 
should  be  aggressively  pursued  and  well  funded  because  the  cost  of  post- 
Milestone  B  problems  dwarfs  the  resources  necessary  to  assure  a  successful 
outcome  from  this  S&T  phase.  The  exit  criteria  established  at  such  early  CDs  for 
subsequent  Milestone  B  consideration  should  take  this  into  account. 

Three  to  four  months  after  a  Concept  Decision  to  proceed,  an  early  TRA  should 
be  completed.  One  important  outcome  of  the  early  TRA  is  a  Technology 
Maturation  Plan  that  should  be  integrated  into  the  Technology  Development 
Strategy  and  should  be  consistent  with  the  Test  and  Evaluation  Strategy  and  the 
Systems  Engineering  Plan  (SEP)  that  will  be  needed  for  Milestone  B.  These 
documents  should  refine  the  preliminary  plans  formulated  during  the  EoA  for 
achieving  the  technical  risk  reduction  necessary  for  Milestone  B  approval.  If  the 
results  of  these  efforts  do  not  validate  the  assumptions  underlying  the  CD,  then 
the  decision-making  body  should  be  reengaged  to  detennine  an  alternate  course  of 
action. 


13  The  Plan  may  be  updated  during  Technology  Development  as  long  as  all  changes  are  coordinated  with 
the  Component  S&T  Executive  and  the  DUSD(S&T). 
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•  Milestone  B 


Feasibility  to  enter  the  acquisition  management  process  at  Milestone  B  implies 
that  the  advancement  degree  of  difficulty  is  low  and  the  technical  suitability  is 
high.  This  is  the  case  when  ( 1)  it  is  clear  that  the  technical  solution  will  satisfy  all 
constraints  and  (2)  the  approach  for  continued  development  should  be  based  on 
having  a  demonstrated  prototype  in  all  areas,  but  some  applications  may  require 
modification  of  that  prototype  for  the  situation  at  hand.  Key  elements  of  the 
development  approach  vary  as  a  function  of  the  technical  risk  areas  as  follows: 

o  For  technology  maturity,  the  development  approach  applies  to  all  critical 
technologies  being  demonstrated  in  a  relevant  environment, 
o  For  test  and  evaluation  readiness,  the  development  approach  applies  to  test 
equipment,  test  tooling,  test  measurement  systems,  software,  . . . 
o  For  design  and  integration  readiness,  the  development  approach  applies  to 
functional  requirements,  interfaces,  constraints,  . . . 
o  For  manufacturing  readiness,  the  development  approach  applies  to 
materials,  machines,  tooling,  metrology,  software,  . . . 
o  For  supportability  readiness,  the  development  approach  applies  to 
maintainability,  reliability,  support  equipment,  logistics  footprint,  . . . 

Entry  into  the  acquisition  management  process  at  Milestone  B  is  contingent  on 
completing  the  statutory  and  regulatory  requirements  that  demonstrate  the 
technical  readiness  to  proceed.  This  includes  the  TRA,  the  Test  and  Evaluation 
Master  Plan  (TEMP),  perhaps  an  updated  AoA,  and  the  SEP.  The  analysis  behind 
and  the  preparation  of  these  documents  must  validate  the  conclusions  reached 
during  the  EoA  concerning  technical  risk.  If  that  is  not  the  case,  and  the  technical 
risk  for  entry  into  Milestone  B  is  too  high,  then  the  request  for  Milestone  B 
approval  should  be  deferred  and  the  decision-making  body  should  be  reengaged 
to  determine  an  alternate  course  of  action.14 

•  Milestone  C 

Feasibility  to  enter  production  at  Milestone  C  implies  that  the  advancement 
degree  of  difficulty  is  extremely  low  and  technical  suitability  is  proven.  This  is 
the  case  when  the  technical  suitability  and  the  approach  for  continued 
development  are  based  on  having  a  demonstrated  production  system  in  all  areas. 


14  With  regard  to  technology  maturity,  to  better  enforce  compliance  with  DoDI  5000.2,  Congress  created 
Section  2366a  of  title  10,  United  States  Code,  as  enacted  by  Section  801  of  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2006  (Public  Law  109-163)  and  as  amended  by  Section  805  of  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2007  (Public  Law  109-365).  This  legislation 
requires  the  Milestone  Decision  Authority  (MDA)  for  a  Major  Defense  Acquisition  Program  to  make 
certain  certifications  concerning  adherence  to  DoD’s  acquisition  policy  prior  to  Milestone  B  or  Key 
Decision  Point  B  approval  of  program  entry  into  System  Development  and  Demonstration.  One  such 
certification  is  that  “the  technology  in  the  program  has  been  demonstrated  in  a  relevant  environment.” 
Another  is  that  “appropriate  market  research  has  been  conducted  prior  to  technology  development  to 
reduce  duplication  of  existing  technology  and  products.”  If  either  condition  is  waived,  the  MDA  must 
provide  the  rationale  to  Congress  based  on  National  Security  concerns. 
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Key  elements  of  the  development  approach  vary  as  a  function  of  the  technical  risk 
areas  as  follows: 


o  For  test  and  evaluation  readiness,  the  development  approach  applies  to 
design  for  testability. 

o  For  design  and  integration  readiness,  the  development  approach  applies 
design  for  producibility. 

o  For  manufacturing  readiness,  the  development  approach  applies  to 
manufacturing  process  flow,  manufacturing  process  variability,  yield 
o  For  supportability  readiness,  the  development  approach  applies  to  design 
for  availability,  life  cycle  cost 

As  was  the  case  for  Milestone  B,  all  statutory  and  regulatory  requirements  must 
be  fulfilled.  These  include  a  TRA,  a  TEMP,  and  a  SEP.  The  analysis  behind  and 
the  preparation  of  these  documents  must  validate  the  conclusions  reached  during 
the  EoA  concerning  technical  risk.  If  that  is  not  the  case,  and  the  technical  risk 
too  high  to  enter  Low  Rate  Initial  Production,  then  the  request  for  Milestone  C 
approval  should  be  deferred  and  the  decision-making  body  should  be  reengaged 
to  determine  an  alternate  course  of  action. 

6.  Summary 

The  CD  process  is  designed  to  accelerate  the  delivery  of  joint  warfighting 
capabilities,  address  resource/funding  constraints  early,  and  tailor  analysis  to  support  the 
decision  at  hand.  To  accomplish  this,  technical  risk  must  be  adequately  addressed  to 
achieve  program  stability  and  predictable  outcomes.  Technical  risk  policy  is  written  for 
the  current  acquisition  management  process  and  must  be  adapted  to  the  new  CD  process. 
The  best  practices  for  doing  this  are  a  function  of  the  type  of  materiel  investment  decision 
being  considered  for  a  Concept  Decision. 

The  EoA  should  assess  the  advancement  degree  of  difficulty  and  the  technical 
suitability  for  the  application  for  every  area  of  technical  risk.  The  results  should  be  used 
to  eliminate  alternatives  where  the  technological  risk  is  too  high  and  to  determine  the 
appropriate  entry  point  into  the  acquisition  framework  for  others.  While  the  EoA  process 
itself  may  be  technically  sufficient  for  the  alternative  to  enter  the  acquisition  management 
framework  at  Milestone  A,  greater  technical  knowledge  obtained  from  a  more  refined 
technical  analysis  is  necessary  before  going  further.  These  analyses  should  be  used  to 
validate  the  EoA  results  before  seeking  a  Milestone  B  or  C  approval. 

Implementing  an  effective  approach  for  monitoring  technology  maturity  in  an 
EoA  will  help  shift  the  focus  of  acquisition  management  from  the  System  Development 
and  Demonstration  Phase  to  the  period  of  time  prior  to  the  formally  defined  “acquisition 
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management”  phase  where  broad  options  are  still  available  and  can  be  aligned  for  closing 
capability  gaps. 
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Appendix  C 

ENABLING  STRATEGIC  CHOICE  THROUGH  INSTITUTIONAL 
REFORM  AND  GOVERNANCE 


QDR  Direction  for  Institutional  Reform  and  Governance 

The  2005  Quadrennial  Defense  Review  (QDR)  provided  the  following  direction 
with  respect  to  Institutional  Refonn  and  Governance  (IR&G):  (1)  Create  or  invigorate 
empowered  horizontal  organizations  to  integrate  priority  areas;  (2)  Improve  Department 
effectiveness  and  efficiency  to  include  exploring  a  portfolio  based  approach  (emphasis 
added)  to  defense  planning,  programming  and  budgeting;  (3)  Move  toward  common  data 
structures/approaches  at  the  enterprise  level;  and  (4)  Implement  new  acquisition  policies, 
procedures  and  processes  for  dramatic  improvements  by  all  measures.  The  Concept 
Decision  (CD)  process  is  included  under  item  (4). 

A  March  15,  2007,  memorandum  (subject:  Institutional  Reform  and  Governance 
Actions  to  Critical  Paths  (ACP))  by  the  Deputy  Secretary  of  Defense  initiates 
implementation  of  the  IR&G  approach.  This  appendix  summarizes  the  Deputy 
Secretary’s  memo  and  its  attachments. 

Summary  of  the  Deputy  Secretary’s  Memorandum  dated  March  17,  2007 

The  memo  states  a  need  to  “move  toward  a  general  management  framework  that 
provides  clear  and  executable  strategic  direction  for  the  current,  mid  and  far  term.”  The 
initiative  is  divided  into  six  ACPs: 

•  ACP  1  -  Strategic  Direction 

•  ACP  2  -  Portfolio  Management 

•  ACP  3  -  Decision  Framework 

•  ACP  4  -  Process 

•  ACP  5  -  Strategic  Resource  Allocation 

•  ACP  6  -  Performance  Assessment  and  Feedback 
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The  IR&G  initiative  appears  to  be  quite  far-reaching  in  its  overall  goal  of 
reforming  strategic  and  fiscal  management  of  DoD.  Related  activities  are  as  follows:  (1) 
Capability  Portfolio  Management  Test  Cases,  (2)  Concept  Decision/Time  Defined 
Acquisition,  (3)  Evaluation  of  Alternatives  (EoA),  (4)  Capital  Accounts,  (5)  “Little  A” 
Acquisition  Reform  Initiatives,  (6)  Data  Transparency  Initiatives  (numerous)  and  (7) 
Joint  Task  Assignment  Process. 


Institutional  Reform  and  Governance  Framework 

At  the  highest  level  IR&G  views  the  following  “Corporate  decision  lanes”  (also 
referred  to  as  “general  management  decision  lanes”  in  the  attachment  to  the  Deputy 
Secretary’s  memo)  as  a  way  to  break  down  departmental  activities  into  more  manageable 
components: 

•  Employ  the  force  (Timeframe:  now) 

Operates  with  what  we  have  available  and  provides  performance  feedback  to 
inform  future  decision  making 

•  Manage  the  force  (Timeframe:  1-3  years) 

Prepares,  supports,  sustains,  and  aligns  what  we  have  to  what  we  need 

•  Develop  the  force  (Timeframe,  >  3years) 

Builds  capability  and  capacity  (DOTMLPF)  to  fulfill  future  joint  customer 
needs  (fits  supply  to  demand) 

•  Corporate  support:  Administration  and  support  activities 

The  following  paragraphs  summarize  the  attachment  to  the  DepSecDef  memo 
(which  describes  the  ACPs  in  detail),  highlighting  in  particular  those  sections  that  have 
an  impact  either  directly  or  indirectly  on  the  CD  process. 

ACP  1  -  Strategic  Direction.  This  ACP  stresses  the  need  to  identify  capability 
gaps  for  near,  mid  and  far  term.  Under  the  heading  “Prioritize  Capability  Gaps  for  FY 
2010,”  “Action  lead”  is  assigned  to  the  JROC  with  a  requirement  to  “report  process”  to 
the  DAWG  in  May  2007.  In  addition,  under  the  heading  “Build  a  Better  Planning 
Guidance  Methodology  for  FY  2010,”  it  is  specified  that  the  “guidance  will  incorporate 
capability  priorities,  gaps,  ...and  define  metrics  to  be  used  by  [the  decision  lanes]  to 
determine  how  well  we  are  executing  our  strategic  plan.”  Action  lead  is  assigned  to 
PDUSD(Policy).  Guidance  is  to  be  published  in  November  2007. 

ACP  2  -  Portfolio  Management.  This  ACP  starts  as  follows: 

Statement  of  Expectations:  There  is  value  added  in  the  ability  to  view 

department  activity  through  a  capability  portfolio  lens.  In  certain  capability  areas 

(those  with  high  joint  value,  but  with  no  natural  owner)  there  is  value  in 

establishing  and  monitoring  an  enterprise  wide  capability  portfolio  proponent. 
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And  continues: 

...the  capability  portfolio  manager  (CPM)  is  responsible  for  advocating  a 
balanced  set  of  capabilities  for  a  defined  capability  portfolio  within  a  given 
resource  constraint.  CPMs  will  provide  portfolio-specific  guidance  consistent 
with  the  Department’s  planning  guidance  to  the  Services  and  Defense  Agencies. 

They  will  also  be  responsible  for  assessing  the  implementation  of  that  guidance 
and  serving  as  an  independent  voice  in  existing  decision  processes.  These 
portfolio  managers  are  afforded  direct  access  to  the  [DAWG,  JROC,  DAB]  and 
other  established  forums  to  raise  portfolio  related  issues  when  necessary. 

It  is  not  specified  by  what  process  such  issues  are  to  be  raised.  The  attachment 
goes  on  to  cite  existing  “Portfolio  Management  Experiments”  for  Joint  Command  and 
Control,  Joint  Net-Centric  Operations,  Battlespace  Awareness,  and  Joint  Logistics  and 
states  that  they  “shall  continue  to  evolve  to  fulfill  this  role.”  It  also  states  that  “The 
determination  to  establish  other  portfolio  managers  will  be  made  on  a  case  by  case  basis, 
once  the  existing  portfolios  are  operating  effectively  in  this  vision.”  There  is  no 
elaboration  as  to  what  this  actually  means  or  by  what  process  it  will  occur. 

The  CPMs  will  also: 

Develop  and  present  a  fiscally  informed  Capability  Portfolio  Strategic  Plan  that 
derives  portfolio  strategic  objectives  from  existing  strategic  guidance,  projected 
capability  mix,  dependencies  with  other  capability  portfolios,  performance 
metrics,  and  actions  -  including  needed  analysis  -  to  meet  objectives  and 
mitigate  risk.  Address  how  changes  to  FY  2009  and  beyond  will  be  developed 
and  proposed  to  better  align  resources  to  the  capability  portfolio  strategic  plan. 

The  CPMs  were  directed  to  present  their  capability  portfolio  strategic  plans  to  the 
DAWG  in  March  2007.  For  the  FY  2009  Program  Review,  the  CPMs  will: 

•  Have  the  authority  to  access  and  work  with  components  to  develop  FY  2009 
change  proposals  to  ensure  alignment  to  the  Department’s  capability  goals  as 
articulated  by  the  capability  portfolio  plan  and  submit  independent  change 
proposals  if  there  is  disagreement  with  the  components. 

•  Present  to  the  DAWG  an  independent  portfolio  assessment  of  the  FY  2009 
Program  and  potential  programmatic  issues  to  be  addressed  in  the  FY  2010 
planning  guidance. 

For  FY  2010,  the  CPMs  will: 

Provide  capability  planning  guidance  consistent  with  the  Capability  Portfolio 
Strategic  Plan  as  part  of  the  FY  2010  planning  guidance.  Propose  other  potential 
CPM  authorities  for  the  FY  2010  Program  Review  cycle.  Present  resource 
requirements  needed  to  efficetively  operate  as  portfolio  managers. 

Timeline:  CPMs  to  present  draft  guidance  and  other  potential  authorities  to  the 
DAWG  in  October  2007. 
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The  next  section  is  explicit  in  reference  to  the  CD  process: 

Capability  Requirements,  Solution  Determination,  Implementation  Monitoring, 

CPM  Performance  Assessment,  CPMs  will: 

•  Work  within  the  concept  decision  process  and  provide  independent  input  to 
Concept  Decision  Tri-chair-USD  (AT&L),  VCJCS,  D,  PA&E,  on  capability 
Issues  as  necessary. 

•  Work  within  the  established  JCIDS  process  to  include  working  with 
Functional  Capabilities  Boards  in  fleshing  out  requirements  and  provide 
independent  input  to  the  JROC  on  capability  issues  as  necessary. 

•  Monitor  implementation  of  existing  programs  from  a  system  of  systems 
perspective  to  ensure  alignment  to  cost,  schedule,  and  portfolio  strategy  and 
provide  an  independent  voice  to  the  appropriate  process  owner  if  performance 
monitoring  indicates  a  deviation  from  accepted  risk  levels. 

•  Assess  their  management  effectiveness  and  impact  on  decision  processes 
against  a  set  of  criteria  established  by  the  Deputy  Secretary  on  an  annual  basis 
and  present  the  results  to  the  DAWG  starting  in  December  2007. 

o  IR&G  will  work  with  CPMs  to  clearly  define  the  Deputy's 
performance 

So  far,  only  four  CPMs  have  been  identified,  and  there  appears  to  be  little  overlap 
with  the  four  CD  pilots.  So  questions  as  to  how  these  interactions  will  occur  do  not 
require  resolution  until  either  a  CD  is  proposed  that  would  fall  into  the  existing  four  CPM 
portfolios,  or  new  CPMs  are  established  that  encompass  an  already-defined  or  to-be- 
established  CD.  When  either  event  occurs,  the  following  questions  will  need  to  be 
addressed: 

•  What  does  it  mean  to  provide  “independent  input?” 

•  How  will  the  CPMs  play  in  the  EoA  process  and  particularly  the  trade-off 
process?  Since  the  DepSecDef  memo  says  the  CPMs  are  to  be  “advocates” 
for  their  portfolio,  they  may  not  be  amenable  to  identifying  tradeoffs  within 
their  portfolios. 

•  The  fourth  bullet  in  the  above  extract  asked  the  CPMs  to  assess  their  own 
management  effectiveness.  By  what  means? 

ACP  3  -  Decision  Framework.  This  ACP  outlines  how  a  new  strategy  and 
resource  decision  process  for  the  Department  might  evolve.  It  starts  with  the  following: 

Statement  of  Expectation:  The  Department  needs  an  overarching  framework  to 
create  a  common  sense  of  value;  allow  strategy  to  outcomes  linkage;  and  enable 
integrated  management  information  and  transparency  across  missions,  functions, 
organizations  and  processes. 

The  ACP  continues  with  an  outline  of  a  portfolio/capability-oriented  process  to 
relate  “current  and  future  operational  goals  to  capabilities  and  link  them  to  core 
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Department  management  processes...”  It  calls  for  the  development  of  “experimental 

portfolio  data  displays  for  the  FY  2009  budget  cycle  to  determine  full-up  implementation 

requirements  approach  for  FY  2010  cycle”  for  presentation  to  the  DAWG.  Next  it 

addresses  the  need  “to  improve  data  integration,  transparency  and  agility”  to  support 

“enterprise  decision  making  and  performance  assessment”  and  “metrics  [to]  measure 

progress  toward  [the]  effort’s  strategic  objectives.”  Finally,  the  ACP  says  to:  “Explore 

Realigning  Department  Activity  by  the  General  Management  Framework”  through: 

...an  external  evaluation  of  the  general  management  framework,  what  its 
implementation  would  look  like,  and  the  impact  it  would  have  on  the  Defense 
organization.  Determine  if  aligning  organizations,  functions  and  processes  in  a 
general  management  framework  could  allow  the  Department  to  streamline— take 
work  and  layers  out— create  better  linkage  of  strategy  to  outcomes,  increase 
transparency  and  accountability,  delegate  authority  and  create  an  efficient 
delivery  of  capability  to  the  joint  warfighter  in  the  near,  mid  and  long  term. 

The  CD  initiative,  if  properly  implemented,  will  contribute  to  the  objective  of  this 
APC  because  a  major  purpose  of  CD  is  to  ensure  that  new  acquisition  program  starts  are 
consistent  with  strategic  priorities. 

ACP  4 — Process.  This  ACP  also  has  a  strong  and  explicit  relationship  with  the 

CD  process.  There  are  two  sub-parts.  The  first  is  to  “Experiment  with  Conducting 

Trades  in  an  Integrated  Capability  Portfolio  Framework.”  (emphasis  added) 

Statement  of  Expectation:  An  established  methodology  should  be  developed  to 
facilitate  trades  and  manage  risk  within  and  across  a  portfolio  construct. 
Instantiation  of  the  methodology  must  be  capable  of  facilitating  multiple 
management  structures— centralized  to  federated. 

This  action  directs  forming: 

...a  team  to  experiment  with  the  concept  of  Integrated  Capability  Portfolio  (ICP) 
decision  management  and  support  methodologies  [and]  address  what  processes, 
incentives,  activities  and  linkages  are  needed  to  manage  cross  portfolio  and 
within  portfolio  trades  in  a  hybrid  enterprise  that  supports  both  Centralized  and 
Federated  management  and  accountability  structures. 

Action  was  assigned  to  USD(AT&L),  PA&E  and  Joint  Staff  (no  lead  was  designated) 
with  a  report  to  be  presented  to  the  DAWG  in  August  2007. 

It  is  noteworthy  that  the  tenn  “integrated  capability  portfolio  framework”  is  not 
precisely  defined.  The  JCA  Reassessment  TOR  (see  Appendix  D  for  reference)  suggests 
the  following  definition:  “Executive-level  management  of  capability  groupings  that 
cover  the  entire  DoD  budget  authority.” 

The  second  action  item  directly  addresses  the  CD  process  (extracted  in  full): 
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Institutionalize  the  Department's  Future  Capability  Investment  Determination 
Process-Concept  Decision 

Statement  of  Expectation:  In  order  to  balance  the  Department's  future  capability 
development  (DOTMLPF)  it  is  necessary  to  put  greater  emphasis  on  early 
evaluation  of  solutions  to  meet  capability  needs  in  terms  of  their  priority, 
affordability,  timing  and  feasibility.  This  is  needed  to  better  guide  decisions  on 
where  to  invest  and  divest,  and  maintain  balanced  portfolios  for  near  and  far  term 
strategic  needs  and  resource  constraints. 

Institutionalize  the  concept  decision  process  by  examining  the  evaluation  of 
solutions  in  terms  of  which  Joint  Capability  Areas  and  strategic  goals  the 
solutions  support,  establishing  criteria  to  measure  process  value,  and  identifying 
implementation  needs. 

•  Action  Lead:  USD  (AT&L) 

•  Supporting  Organizations:  JS/J-8,  PA&E  &  Comptroller 

•  Timeline:  Report  progress,  criteria  to  measure  process  value,  and 
implementation  needs  to  DAWG  in  June  2007  and  December  2007  (Feedback  in 
December  should  also  include  a  recommendation  on  how  best  to  institutionalize 
the  concept  decision  process.) 

This  is  reasonably  consistent  with  current  thinking  on  the  CD  process.  One  difference  is 
the  elevated  role  of  the  Comptroller,  who  has  not  been  proposed  for  a  strong  role  in  the 
CD  process.  One  question  is  what  is  meant  by  “establishing  criteria  to  measure  process 
value.”  Another  is  whether  the  USD(C)  would  have  membership  on  the  Tri-Chair 
(making  it  a  quad-chair). 

ACP  5 — Strategic  Resource  Allocation.  This  ACP  is  concerned  with  the  balance 
between  investments  in  current  versus  future  capabilities.  It  is  suggested  that  separate 
fiscal  guidance  be  issued  for  those  two  timeframes.  USD(AT&L)  and  Comptroller  are  to 
co-lead  a  team  to  further  examine  this  concept  and  provide  recommendations.  This  ACP 
seems  to  be  directed  toward  solving  the  age-old  problem  of  using  procurement  to  pay  for 
exigencies  arising  in  the  operating  accounts  (O&M  and  MilPay).  There  is  not  a  strong 
relationship  with  the  CD  process. 

ACP  6 — Performance  Assessment  and  Feedback.  This  ACP  addresses  the  need 
for  an  improved  process  for  performance  assessment,  both  internally  and  for  external 
reporting  under  the  Government  Performance  and  Results  Act.  There  is  only  a  tangential 
relationship  with  the  CD  process. 
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Appendix  D 

THE  FYDP-JCA  MAPPING 


As  discussed  in  the  main  body  of  this  paper,  the  Joint  Capability  Areas  (JCAs) 
have  recently  been  revised  at  the  Tier  1-3  levels,  and  are  still  under  development  for 
lower  levels.  This  appendix  addresses  the  effort  to  map  the  Future  Years  Defense 
Program  (FYDP)  into  the  original  JCAs  as  they  existed  before  the  recently-completed 
revisions.  The  Deputy  Secretary  of  Defense  asked  the  Director,  Program  Analysis  and 
Evaluation  (PA&E)  to  develop  such  a  mapping. 1 


Figure  D-1.  The  PA&E  FYDP — Capability  Mapping  Structure 

The  methodology  for  the  FDYP-JCA  mapping  is  described  in  the  paper 
referenced  in  the  footnote,  and  the  materials  included  here  are,  by  and  large,  extracted 
from  that  document.  Figure  D-1  is  an  overview  of  the  approach.  By  its  nature,  any  such 
mapping  is  going  to  be  very  gross.  For  example,  there  might  be  highly  important  strategic 
objectives  that  are  limited  in  the  resources  that  can  or  need  to  be  allocated  to  them.  (An 

1  “Mapping  Joint  Capability  Areas  to  Resources,”  Program  Resources  and  Information  Systems 
Management  Division,  Office  of  Program  Analysis  and  Evaluation,  May  2006. 
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example  would  be  the  strategic  triad  during  the  latter  phases  of  the  Cold  War.  Although  it 
was  very  important,  it  didn’t  take  a  large  percentage  of  the  DoD  budget  to  maintain  an 
adequate  capability.)  On  the  other  hand,  relatively  low  priority  activities  in  the  strategic 
sense  will  consume  substantial  resources,  because  even  though  they  are  not  a  high 
strategic  priority,  practical  considerations  still  require  that  they  receive  funding.  Much  of 
defense  infrastructure  falls  into  that  category.  For  these  reasons,  one  should  approach  a 
review  of  a  JCA-FYDP  mapping  with  modest  expectations. 

Another  more  practical  consideration  is  that  the  FYDP  program  element  (PE) 
structure  evolved  over  many  years  to  meet  programmatic  needs  or  other  objectives. 
Unfortunately,  some  of  this  evolution  has  not  been  in  the  direction  that  improves  the 
ability  of  the  structure  to  provide  insights  into  the  programs  being  funded.  Thus,  there 
are  numerous  PEs  that  are  basically  “pots  of  money”  that  the  DoD  Components  allocate 
as  they  deem  appropriate,  and  visibility  is  not  an  objective.  (There  may  be  valid  reasons 
for  some  of  this  obfuscation,  for  example,  in  the  case  of  highly  classified  activities.) 

The  PA&E  document  referenced  above  described  how  the  mapping  was 
established.  A  brief  overview  is  presented  here.  For  those  readers  not  familiar  with  it,  it 
is  necessary  to  describe  how  the  FYDP  data  base  is  constructed. 

Program  Elements.  The  PE  is  the  basic  building  block  of  the  FYDP.  The  PE 
structure  is  specified  by  OSD(PA&E)  for  use  by  the  Defense  Components  (Military 
Departments  and  Defense  Agencies)  in  describing  their  future  years  program  (budget 
year  plus  6)  to  OSD.  The  Components  are  not  required  to  use  the  OSD  PE  structure  for 
their  own  programming.  Typically,  they  have  established  a  mapping  of  the  financial 
structures  they  use  internally  into  the  OSD  PEs  for  reporting  purposes.  This  has  the 
interesting  result  that  inquiries  with  the  Components  as  to  the  contents  of  a  PE  can  be  met 
with  blank  stares,  since  working  level  personnel  are  frequently  not  familiar  with  the 
mapping  the  Component  uses  (or  even  that  there  is  one).  The  PE  has  a  title  and  a 
description,  but  these  frequently  contain  little  or  no  detail  of  any  help.  An  exception  is 
the  RDT&E  PEs,  because  these  are  used  for  Congressional  budget  reporting.  So,  for 
those  PEs  excellent  descriptive  materials  can  normally  be  found.  The  PE  code  is  made 
up  of  seven  characters;  an  example  is  02021 14A,  which  is  entitled  “Light  Divisions.” 
The  initial  “02”  specifies  the  FDYP  “Major  Force  Program”  (in  this  case,  General 
Purpose  Forces)  and  the  last  character  specifies  the  Component — in  this  case  Department 
of  Army.  All  direct  funding  for  the  Army’s  light  divisions  is  rolled  up  into  this  PE. 
There  are  separate  records  for  TOA  (Total  Obligational  Authority — the  programmed 
dollars),  manpower,  and  the  number  of  units  of  this  type  that  are  programmed  over  the 
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FYDP.  There  is  also  a  record  created  for  each  year  of  the  FYDP.  The  PE  funding  is 
further  broken  down  into  Resource  Identification  Codes,  which  can  be  rolled  up  into 
budget  appropriations.  Thus,  a  complete  unique  record  consists  of  five  data  elements: 
The  FYDP  position  (which  FYDP  is  it),  the  PE  code,  the  RIC,  the  Fiscal  Year,  and  the 
amount.  In  addition,  there  are  mappings  established  into  two  other  coding  structures — 
the  Defense  Mission  Categories  (DMC)  and  the  Force  and  Infrastructure  Code  (F&IC). 
As  the  name  suggests,  the  DMCs  assign  each  PE  into  a  stratified  mission  category — the 
top  level  is  the  Major  Force  Program  cited  above.  The  F&IC  is  yet  another  assignment 
that  maps  the  PEs  into  different  force  categories  (e.g.,  operating,  support)  and 
infrastructure  such  as  personnel,  training,  logistics,  etc.  (The  referenced  PA&E 
document  provides  more  detail  on  these  coding  structures.) 


Tier  1  Joint  Capability  Areas 


•  Joint  Force  Generation 

•  Joint  Force  Management 

•  Joint  Battlespace  Awareness 

•  Joint  C2 

•  Joint  Net-Centric  Operations 

•  Joint  Public  Affairs  Operations 

•  Joint  Interagency  /  IGO/  NGO/ 

Coordination 

•  Joint  Protection 

•  Joint  Logistics 

•  Defense  Support  of  Civil 

Authorities 

•  Joint  Homeland  Defense 


•  Joint  Global  Deterrence 

•  Joint  Shaping 

•  Joint  Stability  Operations 

•  Joint  Information  Operations 

•  Joint  Access  &  Access  Denial 

Operations 

•  Joint  Special  Operations  & 

Irregular  Operations 

•  Joint  Land  Operations 

•  Joint  Maritime  /Littoral 

Operations 

•  Joint  Air  Operations 

•  Joint  Space  Operations 

•  Joint  Force  projection 


Figure  D-2.  The  Tier  1  Joint  Capability  Areas 

The  Joint  Capability  Areas.  The  JCAs  were  developed  by  the  Joint  Staff  (J-7)2 3 
over  the  past  several  years,  in  response  to  the  recommendation  of  the  “Aldridge 
commission”  that  the  Department  should  move  more  rapidly  in  the  direction  of 
capability-based  planning.  In  response,  as  part  of  the  Operational  Availability  05  Study, 
the  Joint  Staff  developed  the  JCAs  as  a  tiered  decomposition  of  defense  capabilities. 
When  the  PE-JCA  mapping  was  defined,  there  were  22  Tier  1  capabilities  (listed  in 
Figure  D-2  above)  and  240  Tier  2  capabilities.  It  is  clear  from  a  cursory  look  at  the 
original  Tier  1  and  Tier  2  JCAs  that  there  was  a  large  discrepancy  between  that  construct 


2  See  http://www.dtic.mil/fiitureiointwarfare/cap  areas.htm  for  the  JCA  Lexicon  and  related  documents. 

3  “Joint  Defense  Capabilities  Study:  Improving  DoD  Strategic  Planning,  Resourcing  and  Execution  to 
Satisfy  Joint  Capabilities,”  Honorable  E.C.  Aldridge,  et  al  January  2004. 
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and  the  TOR-stated  goal  of  an  “Executive-level  management  of  capability  groupings  that 
cover  the  entire  DoD  budget  authority”  and  that  are  “mutually  exclusive  and  collectively 
exhaustive  for  all  DoD  Capabilities.” 

The  decomposition  of  the  22  Tier  1  JCAs  into  the  240  “capabilities”  at  the  Tier  2 
level  resulted  in  numerous  Tier  2  JCAs  being  included  under  more  than  one  Tier  1  JCA. 
So  if,  for  example,  the  Department  were  to  resource  by  the  240  Tier  2  JCAs,  they  would 
not  roll  up  cleanly  into  the  Tier  1  JCAs.  A  cursory  examination  of  the  original  Tier  1 
JCAs  indicates  many  interactions.  For  example,  Battlespace  Awareness  and  Command 
and  Control  have  very  substantial  overlaps  with  all  the  operational  JCAs,  such  as  Land 
Operations,  Maritime/Littoral  Operations,  and  Air  Operations. 

The  builders  of  the  FYDP-JCA  mapping  overcame  some  of  the  limitations  of  the 
Tier  1  JCAs  by  bringing  some  Tier  2  JCAs  up  to  the  Tier  1  level  and  adding  some 
categories  needed  for  a  meaningful  mapping.  Figure  D-3  shows  the  way  that  was  done. 
The  result  was  an  ability  to  view  the  resources  along  three  dimensions — “roles,  functions, 
and  effects  targets.” 

The  utility  of  the  JCA-FYDP  mapping  should  be  reassessed  after  it  is  revised  for 
the  new  JCA  structure.  Ultimately,  improvements  in  its  usefulness  will  depend  on  a 
restructuring  and  expansion  of  the  PE  building  blocks  that  would  provide  more  fine¬ 
grained  visibility  into  the  DoD  Components’  programs  and  budgets. 


Range  of  PA&E  FYDP-JCA  Mapping* 


Role 

Operational  Force-wide  Support 


Primary  Operating  Environment 

Multiple  Operating  Environments 
Air 

Air  (Sea  based) 

Land 
Sea 
Space 

Tier  1  Joint  Capability  Area 
i - i  -  Tier  2  Joint  Capability  Area 

*  Extracted  from  “Mapping  Joint  Capability  Areas  to  Resources,"  OSD/DPA&E,  May  2006 

Figure  D-3.  The  Range  of  the  PA&E  FYDP-JCA  Mapping 
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